2024-02-10T05:40:46.826 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:40:46.826 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098 --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --verbose --description orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/orig.config.yaml 2024-02-10T05:40:46.855 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:40:46.935 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098 branch: main description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} email: yweinste@redhat.com first_in_suite: false job_id: '7555098' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDSuDyeUMsUBlYzLQH3vS26Qj+6VwA00H7V7MCI3BzTPM51SbgVUPYfKIj1G45pWjrQRmQZuJ7e7YTEvf85GKY= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBETJdK0fND4MHZCqDjkm30lfLE4kzHagSTV9vkePHnOc9S9KQxA7OCEnMWhnDF2ewnFfpA68J3cfkJpZXoE51E= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T05:40:46.935 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:40:46.936 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:40:46.936 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:40:46.937 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:40:46.961 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:40:46.961 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:40:46.961 INFO:teuthology.packaging:ref: None 2024-02-10T05:40:46.961 INFO:teuthology.packaging:tag: None 2024-02-10T05:40:46.961 INFO:teuthology.packaging:branch: main 2024-02-10T05:40:46.961 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:40:46.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:40:47.213 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:40:47.215 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:40:47.223 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:40:47.223 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:40:47.237 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:40:47.247 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:40:47.258 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:40:47.282 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:35:29.839966', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDSuDyeUMsUBlYzLQH3vS26Qj+6VwA00H7V7MCI3BzTPM51SbgVUPYfKIj1G45pWjrQRmQZuJ7e7YTEvf85GKY='} 2024-02-10T05:40:47.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:35:29.838543', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBETJdK0fND4MHZCqDjkm30lfLE4kzHagSTV9vkePHnOc9S9KQxA7OCEnMWhnDF2ewnFfpA68J3cfkJpZXoE51E='} 2024-02-10T05:40:47.305 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:40:47.316 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T05:40:47.316 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-10T05:40:47.316 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:40:47.392 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f640782d5e0>, signals=[15]) 2024-02-10T05:40:47.393 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:40:47.404 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:40:47.404 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-02-10T05:40:47.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:40:47.495 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-02-10T05:40:47.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:40:47.625 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:40:47.634 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-02-10T05:40:47.651 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-02-10T05:40:47.652 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-02-10T05:40:47.708 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-02-10T05:40:47.708 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-02-10T05:40:47.708 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:40:47.709 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:40:47.710 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-02-10T05:40:47.746 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-02-10T05:40:47.764 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-02-10T05:40:47.764 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-02-10T05:40:47.821 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-02-10T05:40:47.821 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-02-10T05:40:47.821 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-02-10T05:40:47.821 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:40:47.822 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:40:47.822 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-02-10T05:40:47.846 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:40:47.859 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:40:47.873 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:40:47.874 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:40:47.876 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:40:47.892 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:40:47.903 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:40:47.903 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:40:47.933 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:40:47.965 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:40:48.023 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:40:48.043 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:40:48.043 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:40:48.043 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:40:48.044 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:40:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:40:48.059 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-02-10T05:40:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:40:48.060 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-10T05:40:48.115 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:40:48.115 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-02-10T05:40:48.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-02-10T05:40:48.118 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:40:48.118 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-361.el9.x86_64 2024-02-10T05:40:48.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-02-10T05:40:50.206 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-10T05:40:50.483 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-10T05:40:51.773 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-02-10T05:40:52.142 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-02-10T05:40:54.599 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-02-10T05:40:55.889 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-02-10T05:41:02.208 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-02-10T05:41:06.375 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 5.9 MB 00:00 2024-02-10T05:41:07.900 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-02-10T05:41:10.302 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 20 MB 00:02 2024-02-10T05:41:15.226 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-02-10T05:41:17.533 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:41:17.533 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:41:17.632 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:41:17.644 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:41:17.645 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 99 M 2024-02-10T05:41:17.646 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 122 M 2024-02-10T05:41:17.646 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:41:18.328 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 8.8 MB/s | 6.0 MB 00:00 2024-02-10T05:41:19.436 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 22 MB/s | 39 MB 00:01 2024-02-10T05:41:20.536 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 15 MB/s | 33 MB 00:02 2024-02-10T05:41:22.061 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-core-5.14.0-419.el9.x86_64.rpm 4.7 MB/s | 21 MB 00:04 2024-02-10T05:41:22.065 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:41:22.065 INFO:teuthology.orchestra.run.smithi146.stdout:Total 22 MB/s | 99 MB 00:04 2024-02-10T05:41:22.065 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:41:22.436 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:41:22.436 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:41:22.693 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 746 kB/s | 19 MB 00:25 2024-02-10T05:41:23.460 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:41:23.461 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:41:25.871 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:41:26.539 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:41:26.643 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-02-10T05:41:26.669 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:28.194 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-02-10T05:41:29.435 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:29.612 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:34.305 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:34.407 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:41:35.064 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 20 MB 00:06 2024-02-10T05:41:39.134 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:41:39.985 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-02-10T05:41:42.275 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:41:42.276 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:41:42.362 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:41:42.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:41:42.367 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-02-10T05:41:42.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:41:42.369 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 99 M 2024-02-10T05:41:42.369 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 122 M 2024-02-10T05:41:42.369 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:41:42.720 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T05:41:43.220 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-10T05:41:43.878 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 29 MB/s | 33 MB 00:01 2024-02-10T05:41:44.237 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T05:41:44.240 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:41:44.240 INFO:teuthology.orchestra.run.smithi121.stdout:Total 53 MB/s | 99 MB 00:01 2024-02-10T05:41:44.241 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:41:44.623 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:41:44.623 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:41:45.645 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:41:45.645 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:41:48.045 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:41:48.689 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:41:48.820 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:51.574 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:56.471 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:56.584 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:01.317 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:08.203 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:42:08.203 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:42:08.272 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:08.641 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:08.642 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:42:08.642 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:42:08.642 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:42:09.097 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:42:09.627 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:09.802 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:09.803 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:42:09.803 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:42:09.803 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:42:09.803 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T05:42:09.804 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:42:09.804 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:09.911 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:09.912 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:09.912 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-02-10T05:42:09.949 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-02-10T05:42:09.949 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-02-10T05:42:09.949 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-02-10T05:42:09.950 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:42:09.951 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:42:09.952 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:42:09.952 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:42:09.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:42:10.557 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-02-10T05:42:11.951 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:42:11.982 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-02-10T05:42:11.985 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-02-10T05:42:11.999 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.h7fKO5rBiy 2024-02-10T05:42:11.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h7fKO5rBiy 2024-02-10T05:42:12.068 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.h7fKO5rBiy 2024-02-10T05:42:12.209 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.h7fKO5rBiy is 6KB 2024-02-10T05:42:12.258 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.h7fKO5rBiy 2024-02-10T05:42:12.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:42:12.343 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:42:12.344 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:42:12.344 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:42:12.344 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:42:12.345 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:42:12.502 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-02-10T05:42:30.320 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:30.321 INFO:teuthology.orchestra.run.smithi121.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:42:30.321 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:42:30.391 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:42:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:42:30.847 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:42:31.256 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:42:31.257 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:42:31.757 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:31.890 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.891 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:31.891 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:42:31.891 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:42:31.891 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:42:31.891 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T05:42:31.891 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:42:31.891 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:31.999 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:31.999 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:31.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-02-10T05:42:32.041 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-02-10T05:42:32.041 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:42:32.042 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:42:32.043 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:42:32.044 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:42:32.044 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:42:32.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:42:32.694 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-02-10T05:42:34.068 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:42:34.121 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-02-10T05:42:34.124 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-02-10T05:42:34.139 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.YMIfxKlRuz 2024-02-10T05:42:34.139 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YMIfxKlRuz 2024-02-10T05:42:34.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.YMIfxKlRuz 2024-02-10T05:42:34.348 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.YMIfxKlRuz is 6KB 2024-02-10T05:42:34.396 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.YMIfxKlRuz 2024-02-10T05:42:34.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:42:34.475 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:42:34.475 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:42:34.475 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:42:34.475 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:42:34.477 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:42:34.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-02-10T05:42:42.513 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T05:42:42.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:42:42.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:43:04.660 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T05:43:04.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-10T05:43:04.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:43:17.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-02-10T05:43:26.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:43:26.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:43:44.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-02-10T05:43:56.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:43:56.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:43:59.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-02-10T05:44:04.668 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:44:13.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-10T05:44:13.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:44:14.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:44:14.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:44:14.914 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-02-10T05:44:15.306 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:44:15.306 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:44:15.306 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-02-10T05:44:15.322 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:44:15.322 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:44:15.322 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:44:15.322 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:44:16.323 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T05:44:16.323 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:44:16.408 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-02-10T05:44:16.450 DEBUG:teuthology.parallel:result is None 2024-02-10T05:44:17.081 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-10T05:44:17.526 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-10T05:44:17.526 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:44:17.527 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-02-10T05:44:17.540 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:44:17.541 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:44:17.541 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:44:17.541 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:44:18.542 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T05:44:18.542 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:44:18.603 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-02-10T05:44:18.632 DEBUG:teuthology.parallel:result is None 2024-02-10T05:44:18.632 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:44:18.640 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:44:18.640 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:44:18.643 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:44:18.661 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:44:18.683 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:44:18.728 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:44:18.728 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:44:18.731 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:44:18.759 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:44:18.765 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:44:18.765 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:44:18.792 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:44:18.829 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:18.836 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:18.841 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:18.850 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:18.852 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:44:18.859 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:44:18.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:44:18.885 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:44:18.922 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:44:18.931 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:44:18.931 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:18.952 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:18.978 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:19.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:19.099 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:19.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:19.222 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:44:19.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:44:19.294 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:19.337 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:19.395 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:19.485 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:19.523 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:44:19.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:44:19.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-02-10T05:44:19.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-02-10T05:44:19.649 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:44:19.689 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:44:20.188 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:44:20.226 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:44:20.226 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:44:20.306 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:44:20.314 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T05:44:20.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-02-10T05:44:20.402 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-02-10T05:44:20.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-02-10T05:44:20.479 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-02-10T05:44:20.481 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:44:20.481 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-02-10T05:44:20.499 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-02-10T05:44:20.499 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-02-10T05:44:20.524 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-02-10T05:44:20.525 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:44:20.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:44:20.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:44:20.565 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:44:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:44:20.594 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:44:20.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:44:20.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:44:20.810 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:44:20.819 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:44:20.839 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:44:20.840 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T05:48:31.191 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-02-10T05:48:31.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-10T05:48:31.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:31.284 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-02-10T05:48:31.371 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-02-10T05:48:31.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:48:31.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:31.459 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-02-10T05:48:31.547 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-02-10T05:48:31.547 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T05:48:31.558 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T05:48:31.558 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:48:31.558 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:48:31.561 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:48:31.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:48:31.590 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:48:31.607 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:48:31.617 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:48:31.634 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:48:31.637 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-02-10T05:48:31.650 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-02-10T05:48:31.664 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-02-10T05:48:31.667 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:48:31.677 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-02-10T05:48:31.684 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:48:31.694 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:48:31.710 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:48:31.734 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:48:31.737 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:48:31.737 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-02-10T05:48:31.738 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.738 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.738 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.738 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.760 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.764 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:31.764 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T05:48:31.774 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T05:48:31.774 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2024-02-10T05:48:31.775 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T05:48:31.775 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi146.front.sepia.ceph.com 2024-02-10T05:48:31.775 DEBUG:teuthology.orchestra.run.smithi146:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T05:48:32.374 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:48 ago on Sat 10 Feb 2024 05:47:44 AM UTC. 2024-02-10T05:48:32.401 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:49 ago on Sat 10 Feb 2024 05:47:43 AM UTC. 2024-02-10T05:48:32.527 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:48:32.531 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 9.8 M 2024-02-10T05:48:32.532 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:48:32.554 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:48:32.575 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.1 M 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 9.8 M 2024-02-10T05:48:32.576 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:48:32.761 INFO:teuthology.orchestra.run.smithi121.stdout:runc-1.1.12-1.el9.x86_64.rpm 14 MB/s | 3.1 MB 00:00 2024-02-10T05:48:32.762 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:48:32.762 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 3.1 MB 00:00 2024-02-10T05:48:32.762 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:48:32.767 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:48:32.767 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:48:32.835 INFO:teuthology.orchestra.run.smithi146.stdout:runc-1.1.12-1.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-02-10T05:48:32.836 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:48:32.836 INFO:teuthology.orchestra.run.smithi146.stdout:Total 12 MB/s | 3.1 MB 00:00 2024-02-10T05:48:32.836 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:48:32.838 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:48:32.838 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:48:32.841 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:48:32.841 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:48:32.911 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:48:32.912 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:48:33.049 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:48:33.128 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:48:33.148 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:33.229 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:33.713 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:33.809 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:34.027 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:34.027 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:34.027 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:48:34.027 INFO:teuthology.orchestra.run.smithi121.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-10T05:48:34.027 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:34.028 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:48:34.122 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:48:34.290 DEBUG:teuthology.parallel:result is None 2024-02-10T05:48:34.364 DEBUG:teuthology.parallel:result is None 2024-02-10T05:48:34.364 INFO:teuthology.run_tasks:Running task install... 2024-02-10T05:48:34.372 DEBUG:teuthology.task.install:project ceph 2024-02-10T05:48:34.373 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T05:48:34.373 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T05:48:34.373 INFO:teuthology.task.install:Using flavor: default 2024-02-10T05:48:34.379 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T05:48:34.380 INFO:teuthology.task.install:extra packages: [] 2024-02-10T05:48:34.380 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T05:48:34.380 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:48:34.382 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T05:48:34.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:48:34.542 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T05:48:34.543 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T05:48:34.556 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T05:48:34.556 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T05:48:34.671 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T05:48:34.672 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:48:34.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T05:48:34.674 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T05:48:34.674 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:48:34.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T05:48:34.701 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T05:48:34.701 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T05:48:34.703 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T05:48:34.703 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T05:48:34.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T05:48:34.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T05:48:34.849 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T05:48:34.852 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T05:48:34.916 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-02-10T05:48:34.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-02-10T05:48:34.927 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-02-10T05:48:34.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-02-10T05:48:35.154 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-02-10T05:48:35.166 INFO:teuthology.orchestra.run.smithi121.stdout:74 files removed 2024-02-10T05:48:35.180 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw 2024-02-10T05:48:35.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-02-10T05:48:35.845 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 243 kB/s | 82 kB 00:00 2024-02-10T05:48:35.857 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 229 kB/s | 82 kB 00:00 2024-02-10T05:48:36.186 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-02-10T05:48:36.199 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-02-10T05:48:36.537 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-02-10T05:48:36.574 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-02-10T05:48:36.998 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-10T05:48:37.127 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-10T05:48:38.389 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-02-10T05:48:38.612 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-02-10T05:48:39.230 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-10T05:48:39.770 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-02-10T05:48:40.998 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-02-10T05:48:41.723 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-02-10T05:48:44.949 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.9 MB 00:00 2024-02-10T05:48:46.383 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-02-10T05:48:46.730 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-02-10T05:48:47.470 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 21 MB/s | 20 MB 00:00 2024-02-10T05:48:48.114 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-02-10T05:48:52.317 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-10T05:48:55.019 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:48:55.021 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:55.021 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T05:48:55.022 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T05:48:55.023 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-02-10T05:48:55.024 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout:Install 30 Packages 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-02-10T05:48:55.025 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:48:55.026 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 58 M 2024-02-10T05:48:55.026 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:48:56.493 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 4.3 MB/s | 5.3 MB 00:01 2024-02-10T05:48:56.543 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 505 kB/s | 25 kB 00:00 2024-02-10T05:48:56.693 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 4.8 MB/s | 743 kB 00:00 2024-02-10T05:48:56.810 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): libradosstriper1-19.0.0-1361.g290212a1. 4.2 MB/s | 499 kB 00:00 2024-02-10T05:48:57.011 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 5.5 MB/s | 9.7 MB 00:01 2024-02-10T05:48:57.069 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): python3-ceph-argparse-19.0.0-1361.g2902 777 kB/s | 45 kB 00:00 2024-02-10T05:48:57.127 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-ceph-common-19.0.0-1361.g290212 2.2 MB/s | 131 kB 00:00 2024-02-10T05:48:57.186 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): python3-cephfs-19.0.0-1361.g290212a1.el 2.7 MB/s | 161 kB 00:00 2024-02-10T05:48:57.270 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-rados-19.0.0-1361.g290212a1.el9 3.8 MB/s | 319 kB 00:00 2024-02-10T05:48:57.353 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-rbd-19.0.0-1361.g290212a1.el9. 3.5 MB/s | 298 kB 00:00 2024-02-10T05:48:57.412 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-rgw-19.0.0-1361.g290212a1.el9. 1.7 MB/s | 99 kB 00:00 2024-02-10T05:48:57.512 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-02-10T05:48:57.554 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-02-10T05:48:57.588 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-02-10T05:48:57.655 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-02-10T05:48:57.722 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): librgw2-19.0.0-1361.g290212a1.el9.x86_ 5.6 MB/s | 5.1 MB 00:00 2024-02-10T05:48:57.755 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-02-10T05:48:57.772 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-02-10T05:48:57.797 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 5.0 MB/s | 215 kB 00:00 2024-02-10T05:48:57.814 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 809 kB/s | 34 kB 00:00 2024-02-10T05:48:57.831 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-02-10T05:48:58.307 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): ceph-common-19.0.0-1361.g290212a1.el9. 6.5 MB/s | 20 MB 00:03 2024-02-10T05:48:58.607 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 390 kB/s | 309 kB 00:00 2024-02-10T05:48:58.624 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 82 kB/s | 26 kB 00:00 2024-02-10T05:48:58.732 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-02-10T05:48:58.841 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 310 kB/s | 67 kB 00:00 2024-02-10T05:48:59.049 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-02-10T05:48:59.075 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 820 kB/s | 192 kB 00:00 2024-02-10T05:48:59.475 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-02-10T05:48:59.659 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-02-10T05:48:59.751 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): librados2-19.0.0-1361.g290212a1.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-02-10T05:49:00.001 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 6.0 MB/s | 3.1 MB 00:00 2024-02-10T05:49:00.004 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:00.004 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 58 MB 00:04 2024-02-10T05:49:00.004 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:00.040 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:00.040 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:00.526 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:00.526 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:01.936 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:02.015 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T05:49:02.047 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T05:49:02.072 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T05:49:02.095 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T05:49:02.110 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T05:49:02.301 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T05:49:02.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T05:49:02.677 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T05:49:02.740 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:02.832 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:02.921 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T05:49:02.950 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T05:49:03.013 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T05:49:03.042 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T05:49:03.086 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T05:49:03.155 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T05:49:03.178 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T05:49:03.257 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T05:49:03.293 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T05:49:03.401 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T05:49:03.444 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T05:49:03.484 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T05:49:03.517 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T05:49:03.546 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T05:49:03.578 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T05:49:03.903 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T05:49:03.942 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T05:49:04.032 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T05:49:04.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T05:49:04.102 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T05:49:04.576 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T05:49:04.602 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T05:49:04.674 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T05:49:04.749 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T05:49:04.780 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T05:49:06.888 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:06.926 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:07.475 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:07.513 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T05:49:07.583 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T05:49:07.584 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T05:49:07.584 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T05:49:07.584 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:07.622 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:08.813 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:20 2024-02-10T05:49:13.662 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-02-10T05:49:16.353 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:49:16.355 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:16.355 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:49:16.355 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T05:49:16.356 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T05:49:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T05:49:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T05:49:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T05:49:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T05:49:16.357 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T05:49:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T05:49:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T05:49:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T05:49:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T05:49:16.358 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T05:49:16.359 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T05:49:16.360 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout:Install 30 Packages 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 2 Packages 2024-02-10T05:49:16.361 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:16.362 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 58 M 2024-02-10T05:49:16.362 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:49:17.127 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:17.127 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-02-10T05:49:17.127 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-02-10T05:49:17.127 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-02-10T05:49:17.128 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-02-10T05:49:17.128 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-02-10T05:49:17.128 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-02-10T05:49:17.128 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-02-10T05:49:17.128 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:17.722 INFO:teuthology.orchestra.run.smithi146.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 5.3 MB/s | 5.3 MB 00:01 2024-02-10T05:49:17.780 INFO:teuthology.orchestra.run.smithi146.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 433 kB/s | 25 kB 00:00 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi146.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 4.8 MB/s | 743 kB 00:00 2024-02-10T05:49:18.023 INFO:teuthology.orchestra.run.smithi146.stdout:(4/32): libradosstriper1-19.0.0-1361.g290212a1. 5.3 MB/s | 499 kB 00:00 2024-02-10T05:49:18.052 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T05:49:18.076 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T05:49:18.076 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:18.077 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T05:49:18.077 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T05:49:18.077 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T05:49:18.077 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:18.078 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T05:49:18.155 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T05:49:18.155 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:18.156 INFO:teuthology.orchestra.run.smithi146.stdout:(5/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 6.7 MB/s | 9.7 MB 00:01 2024-02-10T05:49:18.240 INFO:teuthology.orchestra.run.smithi146.stdout:(6/32): python3-ceph-argparse-19.0.0-1361.g2902 543 kB/s | 45 kB 00:00 2024-02-10T05:49:18.357 INFO:teuthology.orchestra.run.smithi146.stdout:(7/32): python3-ceph-common-19.0.0-1361.g290212 1.1 MB/s | 131 kB 00:00 2024-02-10T05:49:18.432 INFO:teuthology.orchestra.run.smithi146.stdout:(8/32): python3-cephfs-19.0.0-1361.g290212a1.el 2.1 MB/s | 161 kB 00:00 2024-02-10T05:49:18.540 INFO:teuthology.orchestra.run.smithi146.stdout:(9/32): librgw2-19.0.0-1361.g290212a1.el9.x86_6 9.9 MB/s | 5.1 MB 00:00 2024-02-10T05:49:18.566 INFO:teuthology.orchestra.run.smithi146.stdout:(10/32): python3-rados-19.0.0-1361.g290212a1.el 2.3 MB/s | 319 kB 00:00 2024-02-10T05:49:18.750 INFO:teuthology.orchestra.run.smithi146.stdout:(11/32): ceph-common-19.0.0-1361.g290212a1.el9. 9.8 MB/s | 20 MB 00:02 2024-02-10T05:49:18.774 INFO:teuthology.orchestra.run.smithi146.stdout:(12/32): python3-rbd-19.0.0-1361.g290212a1.el9. 1.2 MB/s | 298 kB 00:00 2024-02-10T05:49:18.800 INFO:teuthology.orchestra.run.smithi146.stdout:(13/32): python3-rgw-19.0.0-1361.g290212a1.el9. 424 kB/s | 99 kB 00:00 2024-02-10T05:49:18.950 INFO:teuthology.orchestra.run.smithi146.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 352 kB/s | 44 kB 00:00 2024-02-10T05:49:18.975 INFO:teuthology.orchestra.run.smithi146.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-02-10T05:49:19.000 INFO:teuthology.orchestra.run.smithi146.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 905 kB/s | 159 kB 00:00 2024-02-10T05:49:19.059 INFO:teuthology.orchestra.run.smithi146.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-02-10T05:49:19.076 INFO:teuthology.orchestra.run.smithi146.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 549 kB/s | 41 kB 00:00 2024-02-10T05:49:19.103 INFO:teuthology.orchestra.run.smithi146.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-02-10T05:49:19.126 INFO:teuthology.orchestra.run.smithi146.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 3.1 MB/s | 215 kB 00:00 2024-02-10T05:49:19.143 INFO:teuthology.orchestra.run.smithi146.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 510 kB/s | 34 kB 00:00 2024-02-10T05:49:19.160 INFO:teuthology.orchestra.run.smithi146.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-02-10T05:49:19.227 INFO:teuthology.orchestra.run.smithi146.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T05:49:19.262 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T05:49:19.263 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T05:49:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T05:49:19.265 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T05:49:19.265 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T05:49:19.265 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:19.265 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T05:49:19.265 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T05:49:19.277 INFO:teuthology.orchestra.run.smithi146.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-02-10T05:49:19.294 INFO:teuthology.orchestra.run.smithi146.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 724 kB/s | 49 kB 00:00 2024-02-10T05:49:19.336 INFO:teuthology.orchestra.run.smithi146.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-02-10T05:49:19.378 INFO:teuthology.orchestra.run.smithi146.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-02-10T05:49:19.512 INFO:teuthology.orchestra.run.smithi146.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-02-10T05:49:19.554 INFO:teuthology.orchestra.run.smithi146.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-02-10T05:49:19.579 INFO:teuthology.orchestra.run.smithi146.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-02-10T05:49:19.954 INFO:teuthology.orchestra.run.smithi146.stdout:(31/32): librados2-19.0.0-1361.g290212a1.el9.x8 7.6 MB/s | 3.4 MB 00:00 2024-02-10T05:49:20.013 INFO:teuthology.orchestra.run.smithi146.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 6.8 MB/s | 3.1 MB 00:00 2024-02-10T05:49:20.016 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:20.016 INFO:teuthology.orchestra.run.smithi146.stdout:Total 16 MB/s | 58 MB 00:03 2024-02-10T05:49:20.017 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:49:20.049 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:49:20.049 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.341 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T05:49:20.342 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:20.343 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:20.526 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:49:20.526 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:49:20.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-02-10T05:49:21.191 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:29 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:21.349 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:21.349 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:21.349 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-02-10T05:49:21.351 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:21.351 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 48 M 2024-02-10T05:49:21.351 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 204 M 2024-02-10T05:49:21.351 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:49:21.534 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-02-10T05:49:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-10T05:49:21.609 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-10T05:49:21.651 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T05:49:21.718 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 446 kB/s | 163 kB 00:00 2024-02-10T05:49:21.942 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:49:22.023 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T05:49:22.059 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T05:49:22.084 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T05:49:22.116 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T05:49:22.131 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T05:49:22.302 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T05:49:22.371 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T05:49:22.699 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T05:49:22.744 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:22.835 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:22.924 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T05:49:22.954 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T05:49:23.008 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T05:49:23.038 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T05:49:23.082 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T05:49:23.150 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T05:49:23.173 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T05:49:23.236 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T05:49:23.263 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T05:49:23.375 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T05:49:23.415 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T05:49:23.455 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T05:49:23.487 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T05:49:23.516 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T05:49:23.549 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T05:49:23.701 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 20 MB/s | 47 MB 00:02 2024-02-10T05:49:23.704 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:23.705 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 48 MB 00:02 2024-02-10T05:49:23.705 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:23.725 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:23.725 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:23.878 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T05:49:23.904 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T05:49:23.999 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T05:49:24.025 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T05:49:24.064 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T05:49:24.125 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:24.125 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:24.515 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T05:49:24.548 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T05:49:24.602 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T05:49:24.669 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T05:49:24.700 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T05:49:24.731 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:24.796 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T05:49:24.826 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T05:49:24.869 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T05:49:24.919 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T05:49:24.946 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T05:49:26.875 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:26.904 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:27.452 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T05:49:27.486 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T05:49:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T05:49:27.580 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T05:49:27.580 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T05:49:27.580 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:27.617 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:29.435 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T05:49:29.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T05:49:32.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T05:49:32.480 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T05:49:32.481 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T05:49:32.481 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:32.481 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:32.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-02-10T05:49:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:41 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:33.511 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T05:49:33.514 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T05:49:33.515 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T05:49:33.516 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout:========================================================================================= 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout:Install 36 Packages 2024-02-10T05:49:33.517 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:33.518 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 30 M 2024-02-10T05:49:33.518 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 107 M 2024-02-10T05:49:33.518 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:49:34.139 INFO:teuthology.orchestra.run.smithi121.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-02-10T05:49:34.348 INFO:teuthology.orchestra.run.smithi121.stdout:(2/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-02-10T05:49:34.381 INFO:teuthology.orchestra.run.smithi121.stdout:(3/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-02-10T05:49:34.473 INFO:teuthology.orchestra.run.smithi121.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1361.g2902 2.6 MB/s | 247 kB 00:00 2024-02-10T05:49:34.582 INFO:teuthology.orchestra.run.smithi121.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-10T05:49:34.624 INFO:teuthology.orchestra.run.smithi121.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-10T05:49:34.658 INFO:teuthology.orchestra.run.smithi121.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-02-10T05:49:34.691 INFO:teuthology.orchestra.run.smithi121.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-10T05:49:34.725 INFO:teuthology.orchestra.run.smithi121.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-10T05:49:34.767 INFO:teuthology.orchestra.run.smithi121.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-10T05:49:34.851 INFO:teuthology.orchestra.run.smithi121.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-02-10T05:49:34.884 INFO:teuthology.orchestra.run.smithi121.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-02-10T05:49:34.918 INFO:teuthology.orchestra.run.smithi121.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-10T05:49:34.951 INFO:teuthology.orchestra.run.smithi121.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-02-10T05:49:34.985 INFO:teuthology.orchestra.run.smithi121.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-02-10T05:49:35.185 INFO:teuthology.orchestra.run.smithi121.stdout:(16/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 20 MB/s | 17 MB 00:00 2024-02-10T05:49:35.202 INFO:teuthology.orchestra.run.smithi121.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-02-10T05:49:35.386 INFO:teuthology.orchestra.run.smithi121.stdout:(18/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-02-10T05:49:35.586 INFO:teuthology.orchestra.run.smithi121.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-02-10T05:49:35.703 INFO:teuthology.orchestra.run.smithi121.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 292 kB/s | 151 kB 00:00 2024-02-10T05:49:35.820 INFO:teuthology.orchestra.run.smithi121.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-02-10T05:49:35.870 INFO:teuthology.orchestra.run.smithi121.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 355 kB/s | 172 kB 00:00 2024-02-10T05:49:35.896 INFO:teuthology.orchestra.run.smithi121.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-02-10T05:49:35.929 INFO:teuthology.orchestra.run.smithi121.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-02-10T05:49:35.979 INFO:teuthology.orchestra.run.smithi121.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-02-10T05:49:36.005 INFO:teuthology.orchestra.run.smithi121.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-02-10T05:49:36.038 INFO:teuthology.orchestra.run.smithi121.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-02-10T05:49:36.097 INFO:teuthology.orchestra.run.smithi121.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 394 kB/s | 46 kB 00:00 2024-02-10T05:49:36.131 INFO:teuthology.orchestra.run.smithi121.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 627 kB/s | 79 kB 00:00 2024-02-10T05:49:36.231 INFO:teuthology.orchestra.run.smithi121.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-02-10T05:49:36.265 INFO:teuthology.orchestra.run.smithi121.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 673 kB/s | 90 kB 00:00 2024-02-10T05:49:36.290 INFO:teuthology.orchestra.run.smithi121.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-02-10T05:49:36.366 INFO:teuthology.orchestra.run.smithi121.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 265 kB/s | 36 kB 00:00 2024-02-10T05:49:36.390 INFO:teuthology.orchestra.run.smithi121.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-02-10T05:49:36.491 INFO:teuthology.orchestra.run.smithi121.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 161 kB/s | 20 kB 00:00 2024-02-10T05:49:36.516 INFO:teuthology.orchestra.run.smithi121.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-02-10T05:49:36.519 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:36.519 INFO:teuthology.orchestra.run.smithi121.stdout:Total 9.9 MB/s | 30 MB 00:03 2024-02-10T05:49:36.538 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:36.583 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:36.583 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:37.000 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:37.001 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-02-10T05:49:37.145 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-02-10T05:49:37.146 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-02-10T05:49:37.146 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-02-10T05:49:37.146 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:38.031 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T05:49:38.055 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T05:49:38.055 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:38.056 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T05:49:38.056 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T05:49:38.056 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T05:49:38.056 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:38.057 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T05:49:38.133 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T05:49:38.133 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:38.301 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:38.393 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T05:49:38.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T05:49:38.471 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T05:49:38.540 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T05:49:38.578 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T05:49:38.610 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T05:49:38.701 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T05:49:38.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T05:49:38.830 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T05:49:38.902 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T05:49:38.955 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T05:49:38.994 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T05:49:39.043 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T05:49:39.097 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T05:49:39.283 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T05:49:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:39.330 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T05:49:39.331 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T05:49:39.332 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T05:49:39.333 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T05:49:39.334 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T05:49:39.369 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T05:49:39.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T05:49:39.435 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T05:49:39.511 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T05:49:39.545 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T05:49:39.579 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T05:49:39.619 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T05:49:39.654 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T05:49:39.697 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T05:49:39.786 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T05:49:39.828 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T05:49:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T05:49:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T05:49:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.313 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T05:49:40.314 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:40.315 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:49:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T05:49:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T05:49:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T05:49:40.375 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:40.376 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T05:49:40.376 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T05:49:40.376 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T05:49:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:40.605 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-test 2024-02-10T05:49:41.180 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:28 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:49:41.338 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:49:41.339 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:41.339 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:49:41.339 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:41.339 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-02-10T05:49:41.340 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:41.341 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 48 M 2024-02-10T05:49:41.341 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 204 M 2024-02-10T05:49:41.341 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:49:41.504 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-02-10T05:49:41.529 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-02-10T05:49:41.571 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-10T05:49:41.613 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T05:49:41.680 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 482 kB/s | 163 kB 00:00 2024-02-10T05:49:41.801 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T05:49:41.827 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:42.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T05:49:42.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:42.502 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T05:49:42.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:42.590 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T05:49:43.963 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 18 MB/s | 47 MB 00:02 2024-02-10T05:49:43.967 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:43.967 INFO:teuthology.orchestra.run.smithi146.stdout:Total 18 MB/s | 48 MB 00:02 2024-02-10T05:49:43.967 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:49:43.987 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:49:43.987 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:49:44.366 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T05:49:44.366 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T05:49:44.367 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T05:49:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T05:49:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T05:49:44.385 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:49:44.385 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:49:44.974 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:49:45.047 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T05:49:45.089 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T05:49:45.149 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T05:49:45.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T05:49:45.217 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T05:49:45.356 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T05:49:45.356 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:45.356 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:49:45.356 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:45.356 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T05:49:45.357 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T05:49:45.358 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:45.359 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:45.735 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-02-10T05:49:46.321 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:54 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:46.426 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:49:46.479 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:46.480 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:49:46.480 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:46.549 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-02-10T05:49:47.135 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:47.291 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:47.292 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:47.292 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:49:47.292 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:47.292 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:49:47.292 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 748 k 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 754 k 2024-02-10T05:49:47.293 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:49:47.604 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 2.3 MB/s | 748 kB 00:00 2024-02-10T05:49:47.605 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:47.605 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.3 MB/s | 748 kB 00:00 2024-02-10T05:49:47.605 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:47.610 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:47.611 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:47.616 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:47.616 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:47.691 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:48.113 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:49:48.193 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:49:48.572 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:49:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:49:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:48.899 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:49:48.899 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:49:48.900 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:48.900 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:49.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T05:49:49.673 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T05:49:49.710 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:57 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:49.748 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T05:49:49.871 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:49.871 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:49.871 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-02-10T05:49:49.871 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 145 k 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 436 k 2024-02-10T05:49:49.872 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:49:50.152 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 521 kB/s | 145 kB 00:00 2024-02-10T05:49:50.152 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:50.153 INFO:teuthology.orchestra.run.smithi121.stdout:Total 518 kB/s | 145 kB 00:00 2024-02-10T05:49:50.153 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:50.160 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:50.160 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:50.212 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:50.212 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:50.353 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:50.428 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:49:50.452 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:49:50.453 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:49:50.453 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T05:49:50.453 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:51.406 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:49:51.406 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:51.406 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:49:51.406 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:51.406 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:51.407 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:51.631 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-02-10T05:49:52.107 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T05:49:52.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T05:49:52.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T05:49:52.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T05:49:52.107 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T05:49:52.108 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T05:49:52.218 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:52.323 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:49:52.375 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:52.376 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:49:52.376 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:49:52.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T05:49:52.555 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T05:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-02-10T05:49:52.556 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:49:52.816 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph 2024-02-10T05:49:53.025 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:01 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:49:53.183 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:49:53.184 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-02-10T05:49:53.184 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:49:53.184 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-02-10T05:49:53.184 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:49:53.184 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T05:49:53.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout:====================================================================================== 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:49:53.186 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.8 M 2024-02-10T05:49:53.187 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-02-10T05:49:53.187 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:49:53.428 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:40 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:49:53.601 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T05:49:53.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T05:49:53.605 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T05:49:53.606 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T05:49:53.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T05:49:53.608 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:========================================================================================= 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:Install 36 Packages 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 30 M 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 107 M 2024-02-10T05:49:53.609 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:49:53.864 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1361.g29 134 kB/s | 28 kB 00:00 2024-02-10T05:49:53.890 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1361.g290 65 kB/s | 15 kB 00:00 2024-02-10T05:49:54.040 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-02-10T05:49:54.073 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-02-10T05:49:54.159 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 3.5 MB/s | 1.8 MB 00:00 2024-02-10T05:49:54.190 INFO:teuthology.orchestra.run.smithi146.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-02-10T05:49:54.441 INFO:teuthology.orchestra.run.smithi146.stdout:(2/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 4.9 MB/s | 2.1 MB 00:00 2024-02-10T05:49:54.525 INFO:teuthology.orchestra.run.smithi146.stdout:(3/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 2.9 MB/s | 1.5 MB 00:00 2024-02-10T05:49:54.641 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-02-10T05:49:54.874 INFO:teuthology.orchestra.run.smithi146.stdout:(4/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86_ 6.8 MB/s | 4.7 MB 00:00 2024-02-10T05:49:54.983 INFO:teuthology.orchestra.run.smithi146.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-10T05:49:55.008 INFO:teuthology.orchestra.run.smithi146.stdout:(6/36): ceph-mgr-modules-core-19.0.0-1361.g2902 513 kB/s | 247 kB 00:00 2024-02-10T05:49:55.033 INFO:teuthology.orchestra.run.smithi146.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-02-10T05:49:55.058 INFO:teuthology.orchestra.run.smithi146.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-02-10T05:49:55.084 INFO:teuthology.orchestra.run.smithi146.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 941 kB/s | 47 kB 00:00 2024-02-10T05:49:55.109 INFO:teuthology.orchestra.run.smithi146.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-02-10T05:49:55.134 INFO:teuthology.orchestra.run.smithi146.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-10T05:49:55.185 INFO:teuthology.orchestra.run.smithi146.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-10T05:49:55.201 INFO:teuthology.orchestra.run.smithi146.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-10T05:49:55.227 INFO:teuthology.orchestra.run.smithi146.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-10T05:49:55.252 INFO:teuthology.orchestra.run.smithi146.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-10T05:49:55.277 INFO:teuthology.orchestra.run.smithi146.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-10T05:49:55.282 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 453 kB/s | 548 kB 00:01 2024-02-10T05:49:55.436 INFO:teuthology.orchestra.run.smithi146.stdout:(17/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 17 MB/s | 17 MB 00:00 2024-02-10T05:49:55.558 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 525 kB/s | 144 kB 00:00 2024-02-10T05:49:55.587 INFO:teuthology.orchestra.run.smithi146.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-02-10T05:49:55.658 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-02-10T05:49:55.811 INFO:teuthology.orchestra.run.smithi146.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 116 kB/s | 43 kB 00:00 2024-02-10T05:49:55.925 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2024-02-10T05:49:56.120 INFO:teuthology.orchestra.run.smithi146.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 179 kB/s | 151 kB 00:00 2024-02-10T05:49:56.237 INFO:teuthology.orchestra.run.smithi146.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2024-02-10T05:49:56.345 INFO:teuthology.orchestra.run.smithi146.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 226 kB/s | 172 kB 00:00 2024-02-10T05:49:56.371 INFO:teuthology.orchestra.run.smithi146.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-02-10T05:49:56.488 INFO:teuthology.orchestra.run.smithi146.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 167 kB/s | 19 kB 00:00 2024-02-10T05:49:56.546 INFO:teuthology.orchestra.run.smithi146.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 488 kB/s | 358 kB 00:00 2024-02-10T05:49:56.614 INFO:teuthology.orchestra.run.smithi146.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2024-02-10T05:49:56.664 INFO:teuthology.orchestra.run.smithi146.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 396 kB/s | 46 kB 00:00 2024-02-10T05:49:56.833 INFO:teuthology.orchestra.run.smithi146.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 359 kB/s | 79 kB 00:00 2024-02-10T05:49:56.950 INFO:teuthology.orchestra.run.smithi146.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2024-02-10T05:49:57.067 INFO:teuthology.orchestra.run.smithi146.stdout:(30/36): python3-jaraco-collections-3.0.0-8.el9 32 kB/s | 23 kB 00:00 2024-02-10T05:49:57.150 INFO:teuthology.orchestra.run.smithi146.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 560 kB/s | 272 kB 00:00 2024-02-10T05:49:57.175 INFO:teuthology.orchestra.run.smithi146.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 399 kB/s | 90 kB 00:00 2024-02-10T05:49:57.267 INFO:teuthology.orchestra.run.smithi146.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 178 kB/s | 36 kB 00:00 2024-02-10T05:49:57.384 INFO:teuthology.orchestra.run.smithi146.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-02-10T05:49:57.425 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 749 kB/s | 2.0 MB 00:02 2024-02-10T05:49:57.534 INFO:teuthology.orchestra.run.smithi146.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 598 kB/s | 230 kB 00:00 2024-02-10T05:49:57.868 INFO:teuthology.orchestra.run.smithi146.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 617 kB/s | 427 kB 00:00 2024-02-10T05:49:57.871 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:57.872 INFO:teuthology.orchestra.run.smithi146.stdout:Total 7.0 MB/s | 30 MB 00:04 2024-02-10T05:49:57.888 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:49:57.933 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:49:57.933 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:49:58.337 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:49:58.337 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:49:58.425 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 197 kB/s | 863 kB 00:04 2024-02-10T05:49:58.431 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:58.431 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.3 MB/s | 6.8 MB 00:05 2024-02-10T05:49:58.451 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:49:58.469 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:49:58.470 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:49:58.668 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:49:58.668 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:49:59.178 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:49:59.258 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T05:49:59.302 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T05:49:59.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T05:49:59.387 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T05:49:59.480 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T05:49:59.611 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T05:49:59.674 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:49:59.698 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T05:49:59.737 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T05:49:59.742 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T05:49:59.773 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T05:49:59.779 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T05:49:59.806 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T05:49:59.809 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T05:49:59.876 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T05:49:59.914 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T05:49:59.946 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T05:50:00.045 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T05:50:00.051 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T05:50:00.086 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T05:50:00.124 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T05:50:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T05:50:00.236 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T05:50:00.291 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T05:50:00.330 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T05:50:00.379 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T05:50:00.459 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T05:50:00.652 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T05:50:00.701 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T05:50:00.738 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T05:50:00.774 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T05:50:00.804 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T05:50:00.889 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T05:50:00.922 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T05:50:00.923 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T05:50:00.931 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T05:50:00.965 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T05:50:01.005 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T05:50:01.039 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T05:50:01.066 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T05:50:01.155 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T05:50:01.205 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T05:50:01.278 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T05:50:01.291 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:01.292 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:01.415 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T05:50:01.540 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T05:50:01.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T05:50:01.707 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T05:50:01.735 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T05:50:01.773 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T05:50:01.773 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:01.774 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T05:50:01.774 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T05:50:01.774 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T05:50:01.774 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:02.125 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:10 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:02.284 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:50:02.285 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-02-10T05:50:02.286 INFO:teuthology.orchestra.run.smithi121.stdout:Install 12 Packages 2024-02-10T05:50:02.287 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:02.287 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 37 M 2024-02-10T05:50:02.287 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 209 M 2024-02-10T05:50:02.287 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:02.690 INFO:teuthology.orchestra.run.smithi121.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-02-10T05:50:02.724 INFO:teuthology.orchestra.run.smithi121.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-02-10T05:50:02.765 INFO:teuthology.orchestra.run.smithi121.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 843 kB/s | 34 kB 00:00 2024-02-10T05:50:02.824 INFO:teuthology.orchestra.run.smithi121.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-10T05:50:03.024 INFO:teuthology.orchestra.run.smithi121.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-02-10T05:50:03.091 INFO:teuthology.orchestra.run.smithi121.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-02-10T05:50:03.108 INFO:teuthology.orchestra.run.smithi121.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-02-10T05:50:03.194 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T05:50:03.221 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:03.317 INFO:teuthology.orchestra.run.smithi121.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 802 kB 00:00 2024-02-10T05:50:03.617 INFO:teuthology.orchestra.run.smithi121.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-02-10T05:50:03.642 INFO:teuthology.orchestra.run.smithi121.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 601 kB/s | 194 kB 00:00 2024-02-10T05:50:03.670 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T05:50:03.693 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:03.912 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T05:50:03.935 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T05:50:03.935 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:03.935 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T05:50:03.936 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T05:50:03.936 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T05:50:03.936 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:04.002 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T05:50:04.451 INFO:teuthology.orchestra.run.smithi121.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 294 kB/s | 245 kB 00:00 2024-02-10T05:50:04.718 INFO:teuthology.orchestra.run.smithi121.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.4 MB/s | 7.4 MB 00:02 2024-02-10T05:50:04.721 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:04.721 INFO:teuthology.orchestra.run.smithi121.stdout:Total 15 MB/s | 37 MB 00:02 2024-02-10T05:50:04.736 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:04.763 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:04.763 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:05.120 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:05.120 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T05:50:05.830 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T05:50:05.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T05:50:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T05:50:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T05:50:05.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T05:50:05.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T05:50:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T05:50:05.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T05:50:05.837 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:05.924 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T05:50:05.955 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T05:50:06.057 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T05:50:06.100 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T05:50:06.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T05:50:06.307 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T05:50:06.384 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T05:50:07.003 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:07.004 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T05:50:07.005 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T05:50:07.006 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:07.007 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:07.054 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T05:50:07.170 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T05:50:07.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-base 2024-02-10T05:50:07.970 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:54 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:08.074 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:08.126 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:08.127 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:08.127 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:08.196 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install cephadm 2024-02-10T05:50:08.417 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T05:50:08.786 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:55 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:08.946 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:08.947 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:08.947 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:08.947 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:08.947 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:08.947 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 748 k 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 754 k 2024-02-10T05:50:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:09.302 INFO:teuthology.orchestra.run.smithi146.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 2.1 MB/s | 748 kB 00:00 2024-02-10T05:50:09.303 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:09.303 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.1 MB/s | 748 kB 00:00 2024-02-10T05:50:09.304 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:09.308 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:09.308 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:09.315 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:09.316 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:09.386 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:09.439 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T05:50:09.466 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T05:50:09.860 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:50:09.940 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:50:10.349 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:50:10.638 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T05:50:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:10.639 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:10.639 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:10.639 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:10.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T05:50:11.429 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:00:58 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:11.590 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:11.591 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 145 k 2024-02-10T05:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 436 k 2024-02-10T05:50:11.592 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:11.840 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 587 kB/s | 145 kB 00:00 2024-02-10T05:50:11.840 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:11.840 INFO:teuthology.orchestra.run.smithi146.stdout:Total 584 kB/s | 145 kB 00:00 2024-02-10T05:50:11.841 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:11.848 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:11.848 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:11.900 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:11.900 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:12.041 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T05:50:12.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T05:50:12.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T05:50:12.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T05:50:12.083 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T05:50:12.099 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:50:12.124 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:50:12.124 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:12.124 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T05:50:12.124 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:12.566 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T05:50:12.566 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:12.567 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:12.980 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-02-10T05:50:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T05:50:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:13.169 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:13.169 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:13.170 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:13.398 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr 2024-02-10T05:50:13.566 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:21 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:13.724 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:13.725 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:50:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:13.725 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:13.725 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T05:50:13.726 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 2.0 M 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 26 M 2024-02-10T05:50:13.727 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:13.997 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:14.104 INFO:teuthology.orchestra.run.smithi146.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:14.156 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:14.158 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:14.158 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:14.226 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-02-10T05:50:14.240 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T05:50:14.250 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 139 kB/s | 24 kB 00:00 2024-02-10T05:50:14.326 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-02-10T05:50:14.351 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-02-10T05:50:14.385 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-02-10T05:50:14.410 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 734 kB/s | 43 kB 00:00 2024-02-10T05:50:14.435 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 145 kB/s | 52 kB 00:00 2024-02-10T05:50:14.686 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-02-10T05:50:14.711 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 99 kB/s | 32 kB 00:00 2024-02-10T05:50:14.833 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:01 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:14.936 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 263 kB/s | 59 kB 00:00 2024-02-10T05:50:14.993 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:14.994 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T05:50:14.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:14.996 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:14.996 INFO:teuthology.orchestra.run.smithi146.stdout:====================================================================================== 2024-02-10T05:50:14.996 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-02-10T05:50:14.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:14.996 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2024-02-10T05:50:14.997 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-02-10T05:50:14.997 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:15.020 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 347 kB/s | 203 kB 00:00 2024-02-10T05:50:15.162 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 398 kB/s | 90 kB 00:00 2024-02-10T05:50:15.660 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1361.g290 73 kB/s | 15 kB 00:00 2024-02-10T05:50:15.686 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1361.g29 121 kB/s | 28 kB 00:00 2024-02-10T05:50:15.821 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 921 kB/s | 1.0 MB 00:01 2024-02-10T05:50:15.822 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:15.823 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.0 MB/s | 2.0 MB 00:02 2024-02-10T05:50:15.823 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:15.828 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-02-10T05:50:15.838 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:15.839 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:15.861 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-02-10T05:50:15.944 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:15.946 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:16.028 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 3.0 MB/s | 1.8 MB 00:00 2024-02-10T05:50:16.187 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-02-10T05:50:16.220 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-02-10T05:50:16.254 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-02-10T05:50:16.271 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-02-10T05:50:16.371 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-02-10T05:50:16.388 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 230 kB/s | 31 kB 00:00 2024-02-10T05:50:16.471 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:16.632 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T05:50:16.696 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T05:50:16.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T05:50:16.779 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-02-10T05:50:16.786 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:16.786 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.8 MB/s | 6.8 MB 00:01 2024-02-10T05:50:16.789 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T05:50:16.804 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:16.821 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:16.821 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:16.830 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T05:50:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T05:50:17.007 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:17.007 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:17.033 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T05:50:17.090 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T05:50:17.521 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:17.588 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T05:50:17.624 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T05:50:17.670 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T05:50:17.700 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T05:50:17.795 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T05:50:17.823 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T05:50:17.871 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T05:50:17.913 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T05:50:17.919 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T05:50:17.961 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T05:50:17.983 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T05:50:18.004 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T05:50:18.047 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T05:50:18.100 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T05:50:18.130 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T05:50:18.372 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T05:50:18.406 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T05:50:18.745 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T05:50:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T05:50:18.746 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T05:50:19.127 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:19.128 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:19.218 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T05:50:19.219 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T05:50:19.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T05:50:19.611 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T05:50:19.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T05:50:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T05:50:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T05:50:19.613 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:19.613 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:19.889 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T05:50:19.959 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:20.120 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:20.120 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:20.120 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:20.121 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================== 2024-02-10T05:50:20.122 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-02-10T05:50:20.122 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:20.122 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 6.8 M 2024-02-10T05:50:20.122 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 33 M 2024-02-10T05:50:20.122 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:20.322 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T05:50:20.467 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:07 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:20.472 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-10T05:50:20.540 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 366 kB/s | 138 kB 00:00 2024-02-10T05:50:20.623 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 388 kB/s | 58 kB 00:00 2024-02-10T05:50:20.626 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:20.627 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-02-10T05:50:20.627 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T05:50:20.628 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout:Install 12 Packages 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:20.629 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 37 M 2024-02-10T05:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 209 M 2024-02-10T05:50:20.630 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:20.681 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-02-10T05:50:20.707 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 308 kB/s | 51 kB 00:00 2024-02-10T05:50:20.711 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:20.711 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 6.8 MB 00:00 2024-02-10T05:50:20.711 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:20.720 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:20.720 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:20.811 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:20.811 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:20.827 INFO:teuthology.orchestra.run.smithi146.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 32 kB 00:00 2024-02-10T05:50:20.852 INFO:teuthology.orchestra.run.smithi146.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 731 kB/s | 17 kB 00:00 2024-02-10T05:50:20.886 INFO:teuthology.orchestra.run.smithi146.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-02-10T05:50:20.945 INFO:teuthology.orchestra.run.smithi146.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-10T05:50:21.114 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:21.128 INFO:teuthology.orchestra.run.smithi146.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-02-10T05:50:21.195 INFO:teuthology.orchestra.run.smithi146.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-02-10T05:50:21.220 INFO:teuthology.orchestra.run.smithi146.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-10T05:50:21.270 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T05:50:21.317 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T05:50:21.362 INFO:teuthology.orchestra.run.smithi146.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-02-10T05:50:21.529 INFO:teuthology.orchestra.run.smithi146.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-10T05:50:21.554 INFO:teuthology.orchestra.run.smithi146.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-10T05:50:21.713 INFO:teuthology.orchestra.run.smithi146.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-02-10T05:50:21.832 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T05:50:21.905 INFO:teuthology.orchestra.run.smithi146.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 6.1 MB/s | 7.4 MB 00:01 2024-02-10T05:50:21.907 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:21.908 INFO:teuthology.orchestra.run.smithi146.stdout:Total 29 MB/s | 37 MB 00:01 2024-02-10T05:50:21.911 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T05:50:21.920 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:21.944 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:21.944 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:21.972 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T05:50:21.996 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T05:50:22.295 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:22.296 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:22.760 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T05:50:22.761 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T05:50:22.761 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T05:50:22.761 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T05:50:22.761 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T05:50:22.761 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T05:50:23.010 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:23.082 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T05:50:23.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:23.082 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:23.082 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:23.083 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:23.094 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T05:50:23.126 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T05:50:23.227 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T05:50:23.262 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T05:50:23.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-02-10T05:50:23.440 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T05:50:23.470 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T05:50:23.547 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T05:50:23.940 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:24.099 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:24.100 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-02-10T05:50:24.101 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:24.101 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 964 k 2024-02-10T05:50:24.101 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 2.8 M 2024-02-10T05:50:24.101 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:24.221 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T05:50:24.225 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 635 kB/s | 79 kB 00:00 2024-02-10T05:50:24.331 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T05:50:24.450 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-02-10T05:50:24.452 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:24.452 INFO:teuthology.orchestra.run.smithi121.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-02-10T05:50:24.452 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:24.472 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:24.473 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:24.550 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:24.550 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:24.740 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:24.868 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T05:50:24.934 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T05:50:25.594 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T05:50:25.624 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T05:50:25.624 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T05:50:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T05:50:25.950 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:25.951 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T05:50:25.951 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:25.951 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:26.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-volume 2024-02-10T05:50:26.615 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T05:50:26.644 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T05:50:26.865 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:34 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:27.025 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-02-10T05:50:27.026 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:27.027 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 332 k 2024-02-10T05:50:27.027 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 1.5 M 2024-02-10T05:50:27.027 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:27.141 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 607 kB/s | 69 kB 00:00 2024-02-10T05:50:27.316 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 915 kB/s | 264 kB 00:00 2024-02-10T05:50:27.317 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:27.317 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-02-10T05:50:27.317 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:27.326 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:27.326 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:27.366 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:27.366 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:27.517 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:27.697 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T05:50:27.727 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T05:50:27.742 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T05:50:27.743 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:27.743 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T05:50:27.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:28.493 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:28.864 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:29.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-02-10T05:50:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T05:50:29.209 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T05:50:29.210 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T05:50:29.619 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T05:50:29.619 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:29.619 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T05:50:29.620 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T05:50:29.621 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T05:50:29.621 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T05:50:29.621 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:29.621 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:29.774 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:37 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:29.938 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-02-10T05:50:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:29.939 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:29.939 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 127 k 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 456 k 2024-02-10T05:50:29.940 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:30.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-rook 2024-02-10T05:50:30.180 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 528 kB/s | 127 kB 00:00 2024-02-10T05:50:30.181 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:30.181 INFO:teuthology.orchestra.run.smithi121.stdout:Total 525 kB/s | 127 kB 00:00 2024-02-10T05:50:30.181 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:30.187 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:30.187 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:30.243 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:30.439 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:30.562 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:30.659 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:17 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:30.819 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T05:50:30.820 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout:Install 13 Packages 2024-02-10T05:50:30.821 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:30.822 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 2.0 M 2024-02-10T05:50:30.822 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 26 M 2024-02-10T05:50:30.822 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:31.024 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:31.094 INFO:teuthology.orchestra.run.smithi146.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2024-02-10T05:50:31.120 INFO:teuthology.orchestra.run.smithi146.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-02-10T05:50:31.170 INFO:teuthology.orchestra.run.smithi146.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-10T05:50:31.195 INFO:teuthology.orchestra.run.smithi146.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-10T05:50:31.221 INFO:teuthology.orchestra.run.smithi146.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T05:50:31.246 INFO:teuthology.orchestra.run.smithi146.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-02-10T05:50:31.271 INFO:teuthology.orchestra.run.smithi146.stdout:(7/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 173 kB/s | 52 kB 00:00 2024-02-10T05:50:31.340 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:31.340 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:31.340 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:31.340 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:31.340 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:31.341 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:31.505 INFO:teuthology.orchestra.run.smithi146.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-02-10T05:50:31.530 INFO:teuthology.orchestra.run.smithi146.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-02-10T05:50:31.565 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-02-10T05:50:31.714 INFO:teuthology.orchestra.run.smithi146.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-02-10T05:50:31.755 INFO:teuthology.orchestra.run.smithi146.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 419 kB/s | 203 kB 00:00 2024-02-10T05:50:31.889 INFO:teuthology.orchestra.run.smithi146.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-02-10T05:50:31.997 INFO:teuthology.orchestra.run.smithi146.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-02-10T05:50:31.999 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:31.999 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-02-10T05:50:32.000 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:32.013 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:32.013 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:32.115 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:32.116 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:32.160 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:32.270 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:32.322 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:32.323 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:32.323 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:32.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-02-10T05:50:32.633 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:32.801 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T05:50:32.877 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T05:50:32.945 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T05:50:32.974 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T05:50:32.979 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:33.016 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T05:50:33.118 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T05:50:33.136 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:33.136 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:33.136 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 k 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 140 k 2024-02-10T05:50:33.137 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:33.235 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T05:50:33.284 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T05:50:33.325 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 166 kB/s | 31 kB 00:00 2024-02-10T05:50:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:Total 165 kB/s | 31 kB 00:00 2024-02-10T05:50:33.326 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:33.329 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:33.329 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:33.353 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:33.354 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:33.447 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:33.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:33.963 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:34.014 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T05:50:34.067 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T05:50:34.098 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T05:50:34.147 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T05:50:34.177 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:34.238 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:34.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-02-10T05:50:34.949 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T05:50:34.949 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T05:50:34.949 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T05:50:34.949 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T05:50:34.950 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T05:50:34.984 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:35.092 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:35.144 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:35.145 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:35.145 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:35.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T05:50:35.334 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:35.335 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:35.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T05:50:35.793 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:35.900 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:35.953 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:35.954 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:35.954 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:36.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-02-10T05:50:36.168 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:36.329 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T05:50:36.330 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout:======================================================================================== 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout:Install 6 Packages 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 6.8 M 2024-02-10T05:50:36.331 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 33 M 2024-02-10T05:50:36.332 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:36.606 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:36.632 INFO:teuthology.orchestra.run.smithi146.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T05:50:36.713 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:36.764 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:36.765 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:36.765 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:36.783 INFO:teuthology.orchestra.run.smithi146.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-10T05:50:36.830 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-02-10T05:50:36.833 INFO:teuthology.orchestra.run.smithi146.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 382 kB/s | 138 kB 00:00 2024-02-10T05:50:37.116 INFO:teuthology.orchestra.run.smithi146.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-02-10T05:50:37.167 INFO:teuthology.orchestra.run.smithi146.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2024-02-10T05:50:37.275 INFO:teuthology.orchestra.run.smithi146.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 854 kB/s | 548 kB 00:00 2024-02-10T05:50:37.277 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:37.277 INFO:teuthology.orchestra.run.smithi146.stdout:Total 7.2 MB/s | 6.8 MB 00:00 2024-02-10T05:50:37.278 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:37.287 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:37.287 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:37.378 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:37.379 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:37.410 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:37.517 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:37.569 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:37.571 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:37.571 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:37.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-02-10T05:50:37.677 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:37.849 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T05:50:37.910 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T05:50:38.219 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:38.327 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:38.379 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:38.380 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:38.380 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:38.433 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T05:50:38.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-02-10T05:50:38.521 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T05:50:38.592 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T05:50:38.622 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T05:50:39.039 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:39.144 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:39.195 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:39.196 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-02-10T05:50:39.196 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:39.264 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T05:50:39.443 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T05:50:39.774 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T05:50:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T05:50:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T05:50:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T05:50:39.775 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:39.775 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:39.845 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:40.006 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:40.006 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 85 k 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 231 k 2024-02-10T05:50:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:40.034 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-fuse 2024-02-10T05:50:40.261 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 336 kB/s | 85 kB 00:00 2024-02-10T05:50:40.262 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:40.262 INFO:teuthology.orchestra.run.smithi121.stdout:Total 334 kB/s | 85 kB 00:00 2024-02-10T05:50:40.262 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:40.269 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:40.269 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:40.324 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:40.324 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:40.613 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:40.773 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:40.774 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:40.775 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-02-10T05:50:40.775 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:40.775 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 964 k 2024-02-10T05:50:40.775 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 2.8 M 2024-02-10T05:50:40.775 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:40.901 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 626 kB/s | 79 kB 00:00 2024-02-10T05:50:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:41.151 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-02-10T05:50:41.153 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:41.153 INFO:teuthology.orchestra.run.smithi146.stdout:Total 2.5 MB/s | 964 kB 00:00 2024-02-10T05:50:41.153 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:41.173 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:41.173 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:41.250 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:41.251 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:41.414 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:41.414 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:41.414 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:41.415 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:41.415 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:41.415 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:41.430 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:41.578 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T05:50:41.642 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T05:50:41.655 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-02-10T05:50:42.252 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:50 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:42.358 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T05:50:42.358 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T05:50:42.414 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.1 M 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 13 M 2024-02-10T05:50:42.415 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:42.692 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:42.923 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 6.1 MB/s | 3.1 MB 00:00 2024-02-10T05:50:42.924 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:42.924 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.1 MB/s | 3.1 MB 00:00 2024-02-10T05:50:42.924 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:42.933 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:42.933 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:43.005 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:43.006 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:43.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-volume 2024-02-10T05:50:43.421 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:43.491 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T05:50:43.516 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:43.655 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:43.818 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T05:50:43.819 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:Install 2 Packages 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 332 k 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 1.5 M 2024-02-10T05:50:43.820 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:43.942 INFO:teuthology.orchestra.run.smithi146.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 564 kB/s | 69 kB 00:00 2024-02-10T05:50:44.105 INFO:teuthology.orchestra.run.smithi146.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 925 kB/s | 264 kB 00:00 2024-02-10T05:50:44.106 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:44.107 INFO:teuthology.orchestra.run.smithi146.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-02-10T05:50:44.107 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:44.115 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:44.116 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:44.164 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:44.164 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:44.320 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:44.505 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T05:50:44.550 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T05:50:44.565 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T05:50:44.565 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:50:44.565 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T05:50:44.566 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:44.644 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:44.644 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:44.645 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:44.645 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:44.645 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:44.645 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:44.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-02-10T05:50:45.316 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T05:50:45.515 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:53 ago on Sat 10 Feb 2024 05:48:52 AM UTC. 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:45.674 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:45.675 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-02-10T05:50:45.675 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:45.675 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-02-10T05:50:45.675 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-02-10T05:50:45.676 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:45.677 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 171 k 2024-02-10T05:50:45.677 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 494 k 2024-02-10T05:50:45.677 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-02-10T05:50:45.944 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 640 kB/s | 171 kB 00:00 2024-02-10T05:50:45.945 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:45.945 INFO:teuthology.orchestra.run.smithi121.stdout:Total 636 kB/s | 171 kB 00:00 2024-02-10T05:50:45.945 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-02-10T05:50:45.953 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-02-10T05:50:45.953 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-02-10T05:50:46.009 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-02-10T05:50:46.009 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-02-10T05:50:46.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados-devel 2024-02-10T05:50:46.156 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-02-10T05:50:46.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:46.648 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:46.703 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:46.809 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T05:50:46.810 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 127 k 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 456 k 2024-02-10T05:50:46.811 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:47.036 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:47.037 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-02-10T05:50:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:47.037 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:50:47.037 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-02-10T05:50:47.186 INFO:teuthology.orchestra.run.smithi146.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 338 kB/s | 127 kB 00:00 2024-02-10T05:50:47.186 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:47.186 INFO:teuthology.orchestra.run.smithi146.stdout:Total 337 kB/s | 127 kB 00:00 2024-02-10T05:50:47.187 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:47.192 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:47.192 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:47.247 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:47.248 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:47.283 DEBUG:teuthology.parallel:result is None 2024-02-10T05:50:47.454 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:47.551 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:47.993 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:48.314 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:48.314 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:48.315 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:48.315 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:48.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs2 2024-02-10T05:50:49.126 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:49.235 INFO:teuthology.orchestra.run.smithi146.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:49.287 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:49.288 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:49.288 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:49.363 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install libcephfs-devel 2024-02-10T05:50:49.947 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:50.104 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:50.105 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:50.106 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 31 k 2024-02-10T05:50:50.106 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 140 k 2024-02-10T05:50:50.106 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:50.438 INFO:teuthology.orchestra.run.smithi146.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 94 kB/s | 31 kB 00:00 2024-02-10T05:50:50.439 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:50.439 INFO:teuthology.orchestra.run.smithi146.stdout:Total 93 kB/s | 31 kB 00:00 2024-02-10T05:50:50.439 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:50.442 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:50.442 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:50.471 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:50.471 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:50.572 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:50.685 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:51.087 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:51.362 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:51.362 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:51.362 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:51.363 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:51.363 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:51.363 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:51.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librados2 2024-02-10T05:50:52.140 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:52.247 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:52.300 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:52.301 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:52.301 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:52.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install librbd1 2024-02-10T05:50:52.956 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:53.064 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:53.116 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:53.117 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:53.117 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:53.195 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rados 2024-02-10T05:50:53.785 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:53.891 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:53.942 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:53.943 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:53.943 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:54.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rgw 2024-02-10T05:50:54.610 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:54.718 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:54.770 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:54.771 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:54.772 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:54.841 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-cephfs 2024-02-10T05:50:55.433 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:55.539 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:55.591 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:55.592 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:55.665 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install python3-rbd 2024-02-10T05:50:56.268 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:56.374 INFO:teuthology.orchestra.run.smithi146.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T05:50:56.424 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:56.425 INFO:teuthology.orchestra.run.smithi146.stdout:Nothing to do. 2024-02-10T05:50:56.425 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:56.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-fuse 2024-02-10T05:50:57.089 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:57.261 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:57.262 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:57.263 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-02-10T05:50:57.263 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:57.263 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:57.263 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T05:50:57.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:57.264 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:57.264 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:57.264 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:57.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:57.264 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 85 k 2024-02-10T05:50:57.265 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 231 k 2024-02-10T05:50:57.265 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:50:57.483 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 389 kB/s | 85 kB 00:00 2024-02-10T05:50:57.484 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:50:57.484 INFO:teuthology.orchestra.run.smithi146.stdout:Total 385 kB/s | 85 kB 00:00 2024-02-10T05:50:57.484 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:50:57.491 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:50:57.492 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:50:57.549 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:50:57.549 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:50:57.686 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:50:57.783 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:58.219 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:58.541 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:50:58.541 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:58.541 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:50:58.541 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:50:58.541 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:58.542 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:50:58.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-mirror 2024-02-10T05:50:59.401 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:50:59.564 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:50:59.564 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:59.564 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repo Size 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:50:59.565 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:50:59.566 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 3.1 M 2024-02-10T05:50:59.566 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 13 M 2024-02-10T05:50:59.566 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:51:00.121 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 5.6 MB/s | 3.1 MB 00:00 2024-02-10T05:51:00.121 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:51:00.122 INFO:teuthology.orchestra.run.smithi146.stdout:Total 5.5 MB/s | 3.1 MB 00:00 2024-02-10T05:51:00.122 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:51:00.131 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:51:00.131 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:51:00.204 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:51:00.205 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:51:00.631 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:51:00.697 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:00.721 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:00.721 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T05:51:00.722 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T05:51:00.722 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T05:51:00.722 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T05:51:00.722 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:01.911 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:01.912 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:51:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:51:01.912 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:01.912 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:51:02.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install rbd-nbd 2024-02-10T05:51:02.750 INFO:teuthology.orchestra.run.smithi146.stdout:Last metadata expiration check: 0:01:49 ago on Sat 10 Feb 2024 05:49:13 AM UTC. 2024-02-10T05:51:02.910 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout:Install 1 Package 2024-02-10T05:51:02.911 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:02.912 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 171 k 2024-02-10T05:51:02.912 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 494 k 2024-02-10T05:51:02.912 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-02-10T05:51:03.157 INFO:teuthology.orchestra.run.smithi146.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 698 kB/s | 171 kB 00:00 2024-02-10T05:51:03.157 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:51:03.158 INFO:teuthology.orchestra.run.smithi146.stdout:Total 694 kB/s | 171 kB 00:00 2024-02-10T05:51:03.158 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-02-10T05:51:03.166 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-02-10T05:51:03.166 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-02-10T05:51:03.221 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-02-10T05:51:03.222 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-02-10T05:51:03.365 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-02-10T05:51:03.472 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:03.946 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:04.245 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T05:51:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:04.246 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-02-10T05:51:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T05:51:04.246 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:51:04.246 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-02-10T05:51:04.495 DEBUG:teuthology.parallel:result is None 2024-02-10T05:51:04.495 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:04.652 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T05:51:04.676 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T05:51:04.677 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T05:51:04.677 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T05:51:04.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:04.839 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T05:51:04.860 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T05:51:04.860 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T05:51:04.860 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T05:51:04.862 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T05:51:04.862 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:51:04.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T05:51:04.895 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:51:04.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T05:51:04.928 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T05:51:04.929 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:51:04.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T05:51:04.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T05:51:05.036 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:51:05.036 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T05:51:05.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T05:51:05.156 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T05:51:05.157 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:51:05.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T05:51:05.187 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T05:51:05.256 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:51:05.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T05:51:05.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T05:51:05.349 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T05:51:05.350 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:51:05.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T05:51:05.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T05:51:05.448 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:51:05.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T05:51:05.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T05:51:05.542 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T05:51:05.641 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'cephadm_mode': 'root'} 2024-02-10T05:51:05.641 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:05.642 INFO:tasks.cephadm:Cluster fsid is 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:05.642 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T05:51:05.642 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.121', 'mon.c': '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]', 'mon.b': '172.21.15.146'} 2024-02-10T05:51:05.642 INFO:tasks.cephadm:First mon is mon.a on smithi121 2024-02-10T05:51:05.642 INFO:tasks.cephadm:First mgr is y 2024-02-10T05:51:05.642 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T05:51:05.643 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-02-10T05:51:05.672 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-02-10T05:51:05.699 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-10T05:51:05.700 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:05.885 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76638/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1361-g290212a1', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1361.g290212a1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-09 18:34:21.832169', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-10T05:51:06.061 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref main, sha1 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:06.062 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm 2024-02-10T05:51:06.062 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:06.548 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:51 /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:06.549 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:06.978 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:51 /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:06.978 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:06.997 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:51:07.025 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on all hosts... 2024-02-10T05:51:07.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:51:07.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:51:07.868 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:51:07.967 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:51:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-02-10T05:51:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-02-10T05:51:43.713 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-02-10T05:51:43.714 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-02-10T05:51:43.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-02-10T05:51:43.775 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-02-10T05:51:43.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-02-10T05:51:43.847 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-02-10T05:51:43.874 INFO:tasks.cephadm:Writing seed config... 2024-02-10T05:51:43.875 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T05:51:43.875 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T05:51:43.875 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T05:51:43.875 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-02-10T05:51:43.875 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-02-10T05:51:43.876 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T05:51:43.877 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:51:43.877 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T05:51:43.904 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 616c67f8-c7d8-11ee-95b8-87774f69a715 mon election default strategy = 3 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-02-10T05:51:43.905 DEBUG:teuthology.orchestra.run.smithi121:mon.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T05:51:43.947 DEBUG:teuthology.orchestra.run.smithi121:mgr.y> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T05:51:43.990 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T05:51:43.990 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -v bootstrap --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.121 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:51:44.186 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:51:44.186 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', '-v', 'bootstrap', '--fsid', '616c67f8-c7d8-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.121', '--skip-admin-label'] 2024-02-10T05:51:44.211 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:51:44.212 INFO:teuthology.orchestra.run.smithi121.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T05:51:44.212 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying podman|docker is present... 2024-02-10T05:51:44.235 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:51:44.235 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying lvm2 is present... 2024-02-10T05:51:44.235 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying time synchronization is in place... 2024-02-10T05:51:44.243 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:51:44.243 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:51:44.249 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:51:44.249 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:51:44.256 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-02-10T05:51:44.263 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-02-10T05:51:44.263 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:51:44.263 INFO:teuthology.orchestra.run.smithi121.stdout:Repeating the final host check... 2024-02-10T05:51:44.285 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:51:44.285 INFO:teuthology.orchestra.run.smithi121.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T05:51:44.286 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl is present 2024-02-10T05:51:44.286 INFO:teuthology.orchestra.run.smithi121.stdout:lvcreate is present 2024-02-10T05:51:44.292 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:51:44.292 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:51:44.298 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:51:44.298 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:51:44.305 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-02-10T05:51:44.312 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-02-10T05:51:44.312 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:51:44.312 INFO:teuthology.orchestra.run.smithi121.stdout:Host looks OK 2024-02-10T05:51:44.312 INFO:teuthology.orchestra.run.smithi121.stdout:Cluster fsid: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:44.313 INFO:teuthology.orchestra.run.smithi121.stdout:Acquiring lock 140669616612976 on /run/cephadm/616c67f8-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:51:44.313 INFO:teuthology.orchestra.run.smithi121.stdout:Lock 140669616612976 acquired on /run/cephadm/616c67f8-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:51:44.313 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 3300 ... 2024-02-10T05:51:44.313 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 6789 ... 2024-02-10T05:51:44.313 INFO:teuthology.orchestra.run.smithi121.stdout:Base mon IP(s) is [172.21.15.121:3300, 172.21.15.121:6789], mon addrv is [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-02-10T05:51:44.316 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.121 metric 100 2024-02-10T05:51:44.316 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.121 metric 100 2024-02-10T05:51:44.319 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T05:51:44.319 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T05:51:44.319 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-02-10T05:51:44.321 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:51:44.322 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-02-10T05:51:44.323 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-02-10T05:51:44.323 INFO:teuthology.orchestra.run.smithi121.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T05:51:44.323 INFO:teuthology.orchestra.run.smithi121.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T05:51:44.323 INFO:teuthology.orchestra.run.smithi121.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:51:44.853 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:51:44.853 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:51:44.853 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T05:51:44.853 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:5fe0e97a267976e17c7dcb240cf9455472de6876101ae78af4bdf182158b6d8d 2024-02-10T05:51:44.854 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T05:51:44.854 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying config sha256:c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:51:44.854 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T05:51:45.792 INFO:teuthology.orchestra.run.smithi121.stdout:ceph: stdout ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:51:45.792 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph version: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:51:45.792 INFO:teuthology.orchestra.run.smithi121.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T05:51:46.813 INFO:teuthology.orchestra.run.smithi121.stdout:stat: stdout 167 167 2024-02-10T05:51:46.813 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial keys... 2024-02-10T05:51:47.818 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQDzDsdlqkFqCRAAhXi8dHNr6vMus3+OynFSrw== 2024-02-10T05:51:48.817 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQD0DsdlgNi8BhAA+q6Y7jvT2YZ3+9AkDeZQUg== 2024-02-10T05:51:49.914 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQD1DsdlzTi5DBAATHRFaV9MgDAUe0oB38jxYA== 2024-02-10T05:51:49.914 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial monmap... 2024-02-10T05:51:50.903 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:51:50.903 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-10T05:51:50.903 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool for a [v2:172.21.15.121:3300,v1:172.21.15.121:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = quincy 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: set fsid to 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:51:50.904 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mon... 2024-02-10T05:51:51.875 INFO:teuthology.orchestra.run.smithi121.stdout:create mon.a on 2024-02-10T05:51:52.064 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-10T05:51:52.202 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T05:51:52.371 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:51:52.371 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:51:52.602 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a 2024-02-10T05:51:52.602 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T05:51:52.762 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:51:53.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:53 smithi121 ceph-mon[29532]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:51:53.282 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:51:53.283 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-02-10T05:51:53.289 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:51:53.289 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:51:53.289 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-02-10T05:51:53.290 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:51:53.290 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon to start... 2024-02-10T05:51:53.290 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon... 2024-02-10T05:51:53.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:53 smithi121 bash[29495]: 38d67bd728baabcf70057b30bcb6a74cb529b95a9a6714175ad2c16400bdcdd5 2024-02-10T05:51:53.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:53 smithi121 systemd[1]: Started Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:51:54.370 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T05:51:54.370 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout id: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:54.370 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout services: 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.571754s) 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout data: 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:54.371 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1492f9b700 1 Processor -- start 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1492f9b700 1 -- start start 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1492f9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1492f9b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c078e50 con 0x7f148c106540 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1490d37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1490d37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40582/0 (socket says 172.21.15.121:40582) 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1490d37700 1 -- 172.21.15.121:0/4136599906 learned_addr learned my addr 172.21.15.121:0/4136599906 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.731+0000 7f1490d37700 1 -- 172.21.15.121:0/4136599906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c108e70 con 0x7f148c106540 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1490d37700 1 --2- 172.21.15.121:0/4136599906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f14800098a0 tx=0x7f1480009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3b833b095c8335 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:54.372 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f148b7fe700 1 -- 172.21.15.121:0/4136599906 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1480004160 con 0x7f148c106540 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f148b7fe700 1 -- 172.21.15.121:0/4136599906 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14800042c0 con 0x7f148c106540 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f148b7fe700 1 -- 172.21.15.121:0/4136599906 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14800045a0 con 0x7f148c106540 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 -- 172.21.15.121:0/4136599906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 msgr2=0x7f148c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 --2- 172.21.15.121:0/4136599906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f14800098a0 tx=0x7f1480009bb0 comp rx=0 tx=0).stop 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 -- 172.21.15.121:0/4136599906 shutdown_connections 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 --2- 172.21.15.121:0/4136599906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c106540 0x7f148c108930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 -- 172.21.15.121:0/4136599906 >> 172.21.15.121:0/4136599906 conn(0x7f148c0fffd0 msgr2=0x7f148c1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 -- 172.21.15.121:0/4136599906 shutdown_connections 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.732+0000 7f1492f9b700 1 -- 172.21.15.121:0/4136599906 wait complete. 2024-02-10T05:51:54.373 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1492f9b700 1 Processor -- start 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1492f9b700 1 -- start start 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1492f9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1492f9b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c199b20 con 0x7f148c1991c0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1490d37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1490d37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40588/0 (socket says 172.21.15.121:40588) 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.733+0000 7f1490d37700 1 -- 172.21.15.121:0/3217484986 learned_addr learned my addr 172.21.15.121:0/3217484986 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1490d37700 1 -- 172.21.15.121:0/3217484986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480009550 con 0x7f148c1991c0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1490d37700 1 --2- 172.21.15.121:0/3217484986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1480008000 tx=0x7f1480004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1480003dc0 con 0x7f148c1991c0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480011630 con 0x7f148c1991c0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c199d20 con 0x7f148c1991c0 2024-02-10T05:51:54.374 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1480011910 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c19c840 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1480022070 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.734+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f148001be20 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.735+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c192d90 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.737+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f148001b450 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.806+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f148c109350 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.807+0000 7f1489ffb700 1 -- 172.21.15.121:0/3217484986 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f148001b5f0 con 0x7f148c1991c0 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 msgr2=0x7f148c1995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:54.375 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 --2- 172.21.15.121:0/3217484986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1480008000 tx=0x7f1480004750 comp rx=0 tx=0).stop 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 shutdown_connections 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 --2- 172.21.15.121:0/3217484986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f148c1991c0 0x7f148c1995e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 >> 172.21.15.121:0/3217484986 conn(0x7f148c0fffd0 msgr2=0x7f148c1003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 shutdown_connections 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:53.808+0000 7f1492f9b700 1 -- 172.21.15.121:0/3217484986 wait complete. 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:mon is available 2024-02-10T05:51:54.376 INFO:teuthology.orchestra.run.smithi121.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T05:51:54.467 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:51:54.467 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: monmap e1: 1 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:54.467 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: fsmap 2024-02-10T05:51:54.468 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:51:54.468 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: mgrmap e1: no daemons active 2024-02-10T05:51:54.468 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:54 smithi121 ceph-mon[29532]: from='client.? 172.21.15.121:0/3217484986' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T05:51:55.473 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:55 smithi121 ceph-mon[29532]: from='client.? 172.21.15.121:0/3793762627' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:51:55.473 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:55 smithi121 ceph-mon[29532]: from='client.? 172.21.15.121:0/3793762627' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:51:55.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:55.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:51:55.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:55.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-02-10T05:51:55.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:51:55.485 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.789+0000 7fac4c349700 1 Processor -- start 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.789+0000 7fac4c349700 1 -- start start 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.789+0000 7fac4c349700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.789+0000 7fac4c349700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac44106480 con 0x7fac44105a90 2024-02-10T05:51:55.486 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac4a0e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac4a0e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40596/0 (socket says 172.21.15.121:40596) 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac4a0e5700 1 -- 172.21.15.121:0/2617164868 learned_addr learned my addr 172.21.15.121:0/2617164868 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac4a0e5700 1 -- 172.21.15.121:0/2617164868 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac44106c90 con 0x7fac44105a90 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac4a0e5700 1 --2- 172.21.15.121:0/2617164868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0098a0 tx=0x7fac3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478f4afc82cc327d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac490e3700 1 -- 172.21.15.121:0/2617164868 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac3c004160 con 0x7fac44105a90 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac490e3700 1 -- 172.21.15.121:0/2617164868 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0042c0 con 0x7fac44105a90 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.790+0000 7fac490e3700 1 -- 172.21.15.121:0/2617164868 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac3c0045a0 con 0x7fac44105a90 2024-02-10T05:51:55.487 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 -- 172.21.15.121:0/2617164868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 msgr2=0x7fac44105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 --2- 172.21.15.121:0/2617164868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0098a0 tx=0x7fac3c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 -- 172.21.15.121:0/2617164868 shutdown_connections 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 --2- 172.21.15.121:0/2617164868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac44105eb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 -- 172.21.15.121:0/2617164868 >> 172.21.15.121:0/2617164868 conn(0x7fac44101090 msgr2=0x7fac441034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 -- 172.21.15.121:0/2617164868 shutdown_connections 2024-02-10T05:51:55.488 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 -- 172.21.15.121:0/2617164868 wait complete. 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.791+0000 7fac4c349700 1 Processor -- start 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4c349700 1 -- start start 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4c349700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4c349700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4407b920 con 0x7fac44105a90 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4a0e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4a0e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40604/0 (socket says 172.21.15.121:40604) 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4a0e5700 1 -- 172.21.15.121:0/3793762627 learned_addr learned my addr 172.21.15.121:0/3793762627 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4a0e5700 1 -- 172.21.15.121:0/3793762627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c009550 con 0x7fac44105a90 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4a0e5700 1 --2- 172.21.15.121:0/3793762627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fac3c004000 tx=0x7fac3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac3c011630 con 0x7fac44105a90 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c011790 con 0x7fac44105a90 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac44079b10 con 0x7fac44105a90 2024-02-10T05:51:55.490 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac3c011a30 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.792+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac44079fb0 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.793+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fac3c022070 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.793+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fac3c01c450 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.793+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4407a9c0 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.795+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fac3c011b90 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.863+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fac441072b0 con 0x7fac44105a90 2024-02-10T05:51:55.491 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.865+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+340 (secure 0 0 0) 0x7fac3c011de0 con 0x7fac44105a90 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.866+0000 7fac3b7fe700 1 -- 172.21.15.121:0/3793762627 <== mon.0 v2:172.21.15.121:3300/0 8 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fac3c01c660 con 0x7fac44105a90 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.866+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 msgr2=0x7fac4407b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.866+0000 7fac4c349700 1 --2- 172.21.15.121:0/3793762627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fac3c004000 tx=0x7fac3c004750 comp rx=0 tx=0).stop 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.867+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 shutdown_connections 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.867+0000 7fac4c349700 1 --2- 172.21.15.121:0/3793762627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac44105a90 0x7fac4407b3e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.867+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 >> 172.21.15.121:0/3793762627 conn(0x7fac44101090 msgr2=0x7fac4418fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.867+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 shutdown_connections 2024-02-10T05:51:55.492 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:54.867+0000 7fac4c349700 1 -- 172.21.15.121:0/3793762627 wait complete. 2024-02-10T05:51:55.493 INFO:teuthology.orchestra.run.smithi121.stdout:Generating new minimal ceph.conf... 2024-02-10T05:51:56.521 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:56 smithi121 ceph-mon[29532]: from='client.? 172.21.15.121:0/2704922146' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:56.522 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f8486700 1 Processor -- start 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f8486700 1 -- start start 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f8486700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f8486700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0105390 con 0x7fe6f01049a0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f6222700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f6222700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40618/0 (socket says 172.21.15.121:40618) 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.920+0000 7fe6f6222700 1 -- 172.21.15.121:0/2152663245 learned_addr learned my addr 172.21.15.121:0/2152663245 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.921+0000 7fe6f6222700 1 -- 172.21.15.121:0/2152663245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0105ba0 con 0x7fe6f01049a0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.921+0000 7fe6f6222700 1 --2- 172.21.15.121:0/2152663245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0098a0 tx=0x7fe6ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2f28d15188d6e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.921+0000 7fe6f5220700 1 -- 172.21.15.121:0/2152663245 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ec004160 con 0x7fe6f01049a0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.921+0000 7fe6f5220700 1 -- 172.21.15.121:0/2152663245 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe6ec0042c0 con 0x7fe6f01049a0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.921+0000 7fe6f5220700 1 -- 172.21.15.121:0/2152663245 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ec0118a0 con 0x7fe6f01049a0 2024-02-10T05:51:56.528 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- 172.21.15.121:0/2152663245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 msgr2=0x7fe6f0104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 --2- 172.21.15.121:0/2152663245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0098a0 tx=0x7fe6ec009bb0 comp rx=0 tx=0).stop 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- 172.21.15.121:0/2152663245 shutdown_connections 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 --2- 172.21.15.121:0/2152663245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f0104dc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- 172.21.15.121:0/2152663245 >> 172.21.15.121:0/2152663245 conn(0x7fe6f00fffa0 msgr2=0x7fe6f01023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- 172.21.15.121:0/2152663245 shutdown_connections 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- 172.21.15.121:0/2152663245 wait complete. 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 Processor -- start 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- start start 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:56.529 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.922+0000 7fe6f8486700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f019c430 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f6222700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f6222700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40620/0 (socket says 172.21.15.121:40620) 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f6222700 1 -- 172.21.15.121:0/2704922146 learned_addr learned my addr 172.21.15.121:0/2704922146 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f6222700 1 -- 172.21.15.121:0/2704922146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec009550 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f6222700 1 --2- 172.21.15.121:0/2704922146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec003f10 tx=0x7fe6ec011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ec0036a0 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fe6ec01a430 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6ec024410 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f019c570 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f019c950 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe6ec024a10 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.923+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe6ec0193c0 con 0x7fe6f01049a0 2024-02-10T05:51:56.530 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.924+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0063350 con 0x7fe6f01049a0 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.928+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe6ec01a5a0 con 0x7fe6f01049a0 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.999+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe6f0108dd0 con 0x7fe6f01049a0 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:55.999+0000 7fe6e77fe700 1 -- 172.21.15.121:0/2704922146 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fe6ec01a8d0 con 0x7fe6f01049a0 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 msgr2=0x7fe6f00792e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 --2- 172.21.15.121:0/2704922146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec003f10 tx=0x7fe6ec011940 comp rx=0 tx=0).stop 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 shutdown_connections 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 --2- 172.21.15.121:0/2704922146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6f01049a0 0x7fe6f00792e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 >> 172.21.15.121:0/2704922146 conn(0x7fe6f00fffa0 msgr2=0x7fe6f0100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 shutdown_connections 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:56.000+0000 7fe6f8486700 1 -- 172.21.15.121:0/2704922146 wait complete. 2024-02-10T05:51:56.531 INFO:teuthology.orchestra.run.smithi121.stdout:Restarting the monitor... 2024-02-10T05:51:56.786 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:56 smithi121 systemd[1]: Stopping Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:51:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:56 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a[29509]: 2024-02-10T05:51:56.784+0000 7f4621de5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:51:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:56 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a[29509]: 2024-02-10T05:51:56.784+0000 7f4621de5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T05:51:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:56 smithi121 podman[29863]: 2024-02-10 05:51:56.795551602 +0000 UTC m=+0.112670543 container died 38d67bd728baabcf70057b30bcb6a74cb529b95a9a6714175ad2c16400bdcdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T05:51:57.403 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29863]: 2024-02-10 05:51:57.115629549 +0000 UTC m=+0.432748487 container cleanup 38d67bd728baabcf70057b30bcb6a74cb529b95a9a6714175ad2c16400bdcdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-10T05:51:57.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 bash[29863]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:51:57.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29882]: 2024-02-10 05:51:57.299782035 +0000 UTC m=+0.500158202 container remove 38d67bd728baabcf70057b30bcb6a74cb529b95a9a6714175ad2c16400bdcdd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T05:51:57.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T05:51:57.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 systemd[1]: Stopped Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:51:57.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 systemd[1]: Starting Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:51:57.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29981]: 2024-02-10 05:51:57.617145467 +0000 UTC m=+0.105542644 container create 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T05:51:57.736 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29981]: 2024-02-10 05:51:57.529239089 +0000 UTC m=+0.017636271 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:51:57.923 INFO:teuthology.orchestra.run.smithi121.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29981]: 2024-02-10 05:51:57.82768702 +0000 UTC m=+0.316084201 container init 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 podman[29981]: 2024-02-10 05:51:57.832788703 +0000 UTC m=+0.321185885 container start 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: pidfile_write: ignore empty --pid-file 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: load: jerasure load: lrc 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Git sha 0 2024-02-10T05:51:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: DB SUMMARY 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: DB Session ID: M58HFRFI5X80P5Q8GIPY 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 86845 ; 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:51:58.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.env: 0x560a2f202ae0 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.info_log: 0x560a30e378c0 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:51:58.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.statistics: (nil) 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.use_fsync: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:51:58.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.db_log_dir: 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.wal_dir: 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.write_buffer_manager: 0x560a30e0b400 2024-02-10T05:51:58.084 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.unordered_write: 0 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:51:58.085 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.row_cache: None 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.wal_filter: None 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.wal_compression: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:51:58.086 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:51:58.087 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_open_files: -1 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Compression algorithms supported: 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kZSTD supported: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:51:58.088 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.merge_operator: 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_filter: None 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:51:58.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560a30e379c0) 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_type: 0 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: data_block_index_type: 0 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_shortening: 1 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: checksum: 4 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: no_block_cache: 0 2024-02-10T05:51:58.090 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache: 0x560a30e3b090 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_options: 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: capacity : 536870912 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: num_shard_bits : 4 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: strict_capacity_limit : 0 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_compressed: (nil) 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: persistent_cache: (nil) 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_size: 4096 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_size_deviation: 10 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_restart_interval: 16 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_block_restart_interval: 1 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: metadata_block_size: 4096 2024-02-10T05:51:58.091 INFO:journalctl@ceph.mon.a.smithi121.stdout: partition_filters: 0 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: use_delta_encoding: 1 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: filter_policy: bloomfilter 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: whole_key_filtering: 1 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: verify_compression: 0 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: format_version: 5 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: enable_index_compression: 1 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_align: 0 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: max_auto_readahead_size: 262144 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: prepopulate_block_cache: 0 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:51:58.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression: NoCompression 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.num_levels: 7 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:51:58.093 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:51:58.094 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:51:58.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:51:58.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:51:58.097 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.ttl: 2592000 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:51:58.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bcfcc1c9-d211-4448-b74b-1dd2cceaf216 2024-02-10T05:51:58.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544317861694, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544317862649, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83267, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 262, "table_properties": {"data_size": 81439, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10592, "raw_average_key_size": 47, "raw_value_size": 75451, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 221, "num_filter_entries": 221, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544317, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bcfcc1c9-d211-4448-b74b-1dd2cceaf216", "db_session_id": "M58HFRFI5X80P5Q8GIPY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544317862729, "job": 1, "event": "recovery_finished"} 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560a30f2c000 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: DB pointer 0x560a30f16000 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** DB Stats ** 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:51:58.100 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** Compaction Stats [default] ** 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: L0 2/0 83.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Sum 2/0 83.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:51:58.101 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** Compaction Stats [default] ** 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:51:58.102 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative compaction: 0.00 GB write, 27.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval compaction: 0.00 GB write, 27.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: Block cache BinnedLRUCache@0x560a30e3b090#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: starting mon.a rank 0 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???) e1 preinit fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:58.103 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).mds e1 new map 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).mds e1 print_map 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: e1 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: legacy client fscid: -1 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout: No filesystems configured 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T05:51:58.104 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 bash[29981]: 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 systemd[1]: Started Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: monmap e1: 1 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: fsmap 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:51:58.105 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:57 smithi121 ceph-mon[30020]: mgrmap e1: no daemons active 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c845a700 1 Processor -- start 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c845a700 1 -- start start 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c845a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c845a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c0106470 con 0x7f50c0105a80 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c61f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c61f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53884/0 (socket says 172.21.15.121:53884) 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.410+0000 7f50c61f6700 1 -- 172.21.15.121:0/3257856169 learned_addr learned my addr 172.21.15.121:0/3257856169 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.411+0000 7f50c61f6700 1 -- 172.21.15.121:0/3257856169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0106c80 con 0x7f50c0105a80 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.411+0000 7f50c61f6700 1 --2- 172.21.15.121:0/3257856169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0098a0 tx=0x7f50bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c8c7176bec5147 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:59.043 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.411+0000 7f50c51f4700 1 -- 172.21.15.121:0/3257856169 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50bc004160 con 0x7f50c0105a80 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.411+0000 7f50c51f4700 1 -- 172.21.15.121:0/3257856169 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f50bc0042c0 con 0x7f50c0105a80 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.411+0000 7f50c51f4700 1 -- 172.21.15.121:0/3257856169 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50bc0118a0 con 0x7f50c0105a80 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- 172.21.15.121:0/3257856169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 msgr2=0x7f50c0105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 --2- 172.21.15.121:0/3257856169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0098a0 tx=0x7f50bc009bb0 comp rx=0 tx=0).stop 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- 172.21.15.121:0/3257856169 shutdown_connections 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 --2- 172.21.15.121:0/3257856169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c0105ea0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- 172.21.15.121:0/3257856169 >> 172.21.15.121:0/3257856169 conn(0x7f50c0101080 msgr2=0x7f50c01034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- 172.21.15.121:0/3257856169 shutdown_connections 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- 172.21.15.121:0/3257856169 wait complete. 2024-02-10T05:51:59.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 Processor -- start 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.412+0000 7f50c845a700 1 -- start start 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c845a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c845a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c007b920 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c61f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c61f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53900/0 (socket says 172.21.15.121:53900) 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c61f6700 1 -- 172.21.15.121:0/4097918037 learned_addr learned my addr 172.21.15.121:0/4097918037 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c61f6700 1 -- 172.21.15.121:0/4097918037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc009550 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c61f6700 1 --2- 172.21.15.121:0/4097918037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f50bc003f10 tx=0x7f50bc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50bc0036a0 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c0079b10 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f50bc01a430 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f50bc024410 con 0x7f50c0105a80 2024-02-10T05:51:59.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c0079fb0 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.413+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f50bc024a60 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.414+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f50bc0193c0 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.414+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c0063350 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.416+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f50bc01a5a0 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.482+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f50c007a2d0 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.484+0000 7f50b77fe700 1 -- 172.21.15.121:0/4097918037 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f50bc01a8d0 con 0x7f50c0105a80 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 msgr2=0x7f50c007b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 --2- 172.21.15.121:0/4097918037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f50bc003f10 tx=0x7f50bc011940 comp rx=0 tx=0).stop 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 shutdown_connections 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 --2- 172.21.15.121:0/4097918037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c0105a80 0x7f50c007b3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 >> 172.21.15.121:0/4097918037 conn(0x7f50c0101080 msgr2=0x7f50c0101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 shutdown_connections 2024-02-10T05:51:59.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:51:58.485+0000 7f50c845a700 1 -- 172.21.15.121:0/4097918037 wait complete. 2024-02-10T05:51:59.047 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T05:51:59.066 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:51:59.067 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mgr... 2024-02-10T05:51:59.067 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-10T05:51:59.067 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-10T05:51:59.323 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y 2024-02-10T05:51:59.323 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T05:51:59.465 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:51:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:51:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4097918037' entity='client.admin' 2024-02-10T05:51:59.828 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:51:59 smithi121 podman[30265]: 2024-02-10 05:51:59.599467669 +0000 UTC m=+0.017737496 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:52:00.076 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:52:00.077 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-02-10T05:52:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:52:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:52:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-02-10T05:52:00.083 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:52:00.091 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:52:00.091 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-02-10T05:52:00.098 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:52:00.098 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:52:00.098 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-02-10T05:52:00.099 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-10T05:52:00.099 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr to start... 2024-02-10T05:52:00.099 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr... 2024-02-10T05:52:00.222 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:51:59 smithi121 podman[30265]: 2024-02-10 05:51:59.95495075 +0000 UTC m=+0.373220572 container init d3fdc87b1771ff349ba201da7601f37107503da602f39a2ce48d5f784dd1e0c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T05:52:00.223 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:51:59 smithi121 podman[30265]: 2024-02-10 05:51:59.960025546 +0000 UTC m=+0.378295358 container start d3fdc87b1771ff349ba201da7601f37107503da602f39a2ce48d5f784dd1e0c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:52:00.223 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:00 smithi121 bash[30265]: d3fdc87b1771ff349ba201da7601f37107503da602f39a2ce48d5f784dd1e0c4 2024-02-10T05:52:00.223 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:00 smithi121 systemd[1]: Started Ceph mgr.y for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:52:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1593667382' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:52:00.828 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:00 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:00.508+0000 7f0168ddf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:52:00.828 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:00 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:00.606+0000 7f0168ddf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:52:01.162 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "616c67f8-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:52:01.165 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-10T05:52:01.166 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:52:01.167 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:52:01.176 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:52:01.176 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:52:01.176 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:52:01.176 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:52:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:01.178 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:51:53.237561+0000", 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f41667ba700 1 Processor -- start 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f41667ba700 1 -- start start 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f41667ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f41667ba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4160078e10 con 0x7f4160104370 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f415ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f415ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53916/0 (socket says 172.21.15.121:53916) 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.586+0000 7f415ffff700 1 -- 172.21.15.121:0/538725192 learned_addr learned my addr 172.21.15.121:0/538725192 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f415ffff700 1 -- 172.21.15.121:0/538725192 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4160104cd0 con 0x7f4160104370 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f415ffff700 1 --2- 172.21.15.121:0/538725192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f41500098a0 tx=0x7f4150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582a5dca2a3f1d12 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:01.179 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f415effd700 1 -- 172.21.15.121:0/538725192 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4150004160 con 0x7f4160104370 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f415effd700 1 -- 172.21.15.121:0/538725192 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f41500042c0 con 0x7f4160104370 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f415effd700 1 -- 172.21.15.121:0/538725192 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41500118a0 con 0x7f4160104370 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f41667ba700 1 -- 172.21.15.121:0/538725192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 msgr2=0x7f4160104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.587+0000 7f41667ba700 1 --2- 172.21.15.121:0/538725192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f41500098a0 tx=0x7f4150009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 -- 172.21.15.121:0/538725192 shutdown_connections 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 --2- 172.21.15.121:0/538725192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f4160104790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 -- 172.21.15.121:0/538725192 >> 172.21.15.121:0/538725192 conn(0x7f41600fffd0 msgr2=0x7f41601023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 -- 172.21.15.121:0/538725192 shutdown_connections 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 -- 172.21.15.121:0/538725192 wait complete. 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 Processor -- start 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.588+0000 7f41667ba700 1 -- start start 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f41667ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:01.180 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f41667ba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416019be90 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53930/0 (socket says 172.21.15.121:53930) 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415ffff700 1 -- 172.21.15.121:0/1593667382 learned_addr learned my addr 172.21.15.121:0/1593667382 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415ffff700 1 -- 172.21.15.121:0/1593667382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4150009550 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415ffff700 1 --2- 172.21.15.121:0/1593667382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4150003da0 tx=0x7f4150011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4150011930 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4150003b00 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416019c090 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f415001ad50 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.589+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416019c530 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.590+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f415002d7d0 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.590+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4150024730 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.590+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41601955f0 con 0x7f4160104370 2024-02-10T05:52:01.181 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.592+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4150019db0 con 0x7f4160104370 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.665+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4160105ba0 con 0x7f4160104370 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.665+0000 7f415d7fa700 1 -- 172.21.15.121:0/1593667382 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f415001a630 con 0x7f4160104370 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 msgr2=0x7f416019b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 --2- 172.21.15.121:0/1593667382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4150003da0 tx=0x7f4150011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 shutdown_connections 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 --2- 172.21.15.121:0/1593667382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4160104370 0x7f416019b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 >> 172.21.15.121:0/1593667382 conn(0x7f41600fffd0 msgr2=0x7f4160192730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 shutdown_connections 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:00.666+0000 7f41667ba700 1 -- 172.21.15.121:0/1593667382 wait complete. 2024-02-10T05:52:01.182 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (1/15)... 2024-02-10T05:52:01.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.259+0000 7f0168ddf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:52:01.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.352+0000 7f0168ddf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:52:01.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.490+0000 7f0168ddf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:52:01.932 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.585+0000 7f0168ddf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:52:01.932 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.667+0000 7f0168ddf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:52:02.215 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:01 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:01.931+0000 7f0168ddf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:52:02.216 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:02 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:02.039+0000 7f0168ddf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:02.216 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:02 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:02.123+0000 7f0168ddf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:52:02.216 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:02 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:02.215+0000 7f0168ddf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:52:02.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:02 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:02.387+0000 7f0168ddf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:52:03.564 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:03 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:03.345+0000 7f0168ddf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:52:03.564 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:03 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:03.478+0000 7f0168ddf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:52:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1565133530' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:52:04.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:03 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:03.824+0000 7f0168ddf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:52:04.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:03 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:03.915+0000 7f0168ddf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "616c67f8-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:52:04.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:52:04.314 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:52:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:52:04.315 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:52:04.323 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:52:04.324 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:51:53.237561+0000", 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08dc4df700 1 Processor -- start 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08dc4df700 1 -- start start 2024-02-10T05:52:04.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08dc4df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08dc4df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d40796e0 con 0x7f08d4106550 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08da27b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08da27b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53932/0 (socket says 172.21.15.121:53932) 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.667+0000 7f08da27b700 1 -- 172.21.15.121:0/3417888820 learned_addr learned my addr 172.21.15.121:0/3417888820 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.668+0000 7f08da27b700 1 -- 172.21.15.121:0/3417888820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d4108e80 con 0x7f08d4106550 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.668+0000 7f08da27b700 1 --2- 172.21.15.121:0/3417888820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08d00098a0 tx=0x7f08d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb86be76c86d1302 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.668+0000 7f08d9279700 1 -- 172.21.15.121:0/3417888820 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08d0004160 con 0x7f08d4106550 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.668+0000 7f08d9279700 1 -- 172.21.15.121:0/3417888820 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f08d00042c0 con 0x7f08d4106550 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.668+0000 7f08d9279700 1 -- 172.21.15.121:0/3417888820 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08d00118a0 con 0x7f08d4106550 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- 172.21.15.121:0/3417888820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 msgr2=0x7f08d4108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 --2- 172.21.15.121:0/3417888820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f08d00098a0 tx=0x7f08d0009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- 172.21.15.121:0/3417888820 shutdown_connections 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 --2- 172.21.15.121:0/3417888820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d4108940 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:04.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- 172.21.15.121:0/3417888820 >> 172.21.15.121:0/3417888820 conn(0x7f08d40fffa0 msgr2=0x7f08d41023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- 172.21.15.121:0/3417888820 shutdown_connections 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- 172.21.15.121:0/3417888820 wait complete. 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 Processor -- start 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.669+0000 7f08dc4df700 1 -- start start 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08dc4df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08dc4df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d4199c20 con 0x7f08d4106550 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08da27b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08da27b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53936/0 (socket says 172.21.15.121:53936) 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08da27b700 1 -- 172.21.15.121:0/1565133530 learned_addr learned my addr 172.21.15.121:0/1565133530 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08da27b700 1 -- 172.21.15.121:0/1565133530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009550 con 0x7f08d4106550 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08da27b700 1 --2- 172.21.15.121:0/1565133530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08d0003da0 tx=0x7f08d0011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08d0011930 con 0x7f08d4106550 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d4199e20 con 0x7f08d4106550 2024-02-10T05:52:04.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f08d0003b00 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f08d001ad50 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.670+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d419a240 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.671+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f08d002d7d0 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.671+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f08d0024730 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.671+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d4109b60 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.675+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f08d0019db0 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.748+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f08d4104490 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.748+0000 7f08cb7fe700 1 -- 172.21.15.121:0/1565133530 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f08d0035090 con 0x7f08d4106550 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 msgr2=0x7f08d41996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 --2- 172.21.15.121:0/1565133530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08d0003da0 tx=0x7f08d0011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 shutdown_connections 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 --2- 172.21.15.121:0/1565133530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08d4106550 0x7f08d41996e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 >> 172.21.15.121:0/1565133530 conn(0x7f08d40fffa0 msgr2=0x7f08d4100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:04.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.749+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 shutdown_connections 2024-02-10T05:52:04.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:03.750+0000 7f08dc4df700 1 -- 172.21.15.121:0/1565133530 wait complete. 2024-02-10T05:52:04.329 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (2/15)... 2024-02-10T05:52:04.574 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:04 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:04.109+0000 7f0168ddf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:04.574 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:04 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:04.291+0000 7f0168ddf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:52:04.828 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:04 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:04.573+0000 7f0168ddf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:04 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:04.829+0000 7f0168ddf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:05.064+0000 7f0168ddf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:52:06.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:05.850+0000 7f0168ddf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:52:06.730 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:06.419+0000 7f0168ddf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:52:06.730 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:06.520+0000 7f0168ddf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:52:06.730 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:06.729+0000 7f0168ddf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: Activating manager daemon y 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: mgrmap e2: y(active, starting, since 0.00286944s) 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:52:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: Manager daemon y is now available 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' 2024-02-10T05:52:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:06 smithi121 ceph-mon[30020]: from='mgr.14100 172.21.15.121:0/3360061783' entity='mgr.y' 2024-02-10T05:52:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:08 smithi121 ceph-mon[30020]: mgrmap e3: y(active, since 1.00655s) 2024-02-10T05:52:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:10 smithi121 ceph-mon[30020]: mgrmap e4: y(active, since 3s) 2024-02-10T05:52:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2591325462' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:52:15.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:15.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "616c67f8-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:52:15.983 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:52:15.984 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.985 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.986 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:51:53.237561+0000", 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.083+0000 7f37887f6700 1 Processor -- start 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.083+0000 7f37887f6700 1 -- start start 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.083+0000 7f37887f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.083+0000 7f37887f6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3780078e50 con 0x7f3780106540 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.083+0000 7f3786592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3786592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58376/0 (socket says 172.21.15.121:58376) 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3786592700 1 -- 172.21.15.121:0/1841711323 learned_addr learned my addr 172.21.15.121:0/1841711323 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:15.987 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3786592700 1 -- 172.21.15.121:0/1841711323 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780108e70 con 0x7f3780106540 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3786592700 1 --2- 172.21.15.121:0/1841711323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098a0 tx=0x7f377c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc678c57f2ecb7e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3785590700 1 -- 172.21.15.121:0/1841711323 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c004160 con 0x7f3780106540 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3785590700 1 -- 172.21.15.121:0/1841711323 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f377c0042c0 con 0x7f3780106540 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.084+0000 7f3785590700 1 -- 172.21.15.121:0/1841711323 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c0118a0 con 0x7f3780106540 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- 172.21.15.121:0/1841711323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 msgr2=0x7f3780108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 --2- 172.21.15.121:0/1841711323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098a0 tx=0x7f377c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- 172.21.15.121:0/1841711323 shutdown_connections 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 --2- 172.21.15.121:0/1841711323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780108930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- 172.21.15.121:0/1841711323 >> 172.21.15.121:0/1841711323 conn(0x7f37800fffd0 msgr2=0x7f37801023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- 172.21.15.121:0/1841711323 shutdown_connections 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- 172.21.15.121:0/1841711323 wait complete. 2024-02-10T05:52:15.988 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 Processor -- start 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.085+0000 7f37887f6700 1 -- start start 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37887f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37887f6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3780199c40 con 0x7f3780106540 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f3786592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f3786592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58384/0 (socket says 172.21.15.121:58384) 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f3786592700 1 -- 172.21.15.121:0/2591325462 learned_addr learned my addr 172.21.15.121:0/2591325462 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f3786592700 1 -- 172.21.15.121:0/2591325462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c009550 con 0x7f3780106540 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f3786592700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f377c003da0 tx=0x7f377c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c011930 con 0x7f3780106540 2024-02-10T05:52:15.989 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f377c003b00 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c01ad50 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3780199e40 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.086+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378019a2e0 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.087+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45921+0+0 (secure 0 0 0) 0x7f377c02d440 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.087+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3780109b50 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.087+0000 7f37777fe700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 0x7f376c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.087+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f377c04cff0 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.087+0000 7f3785d91700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 0x7f376c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.088+0000 7f3785d91700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 0x7f376c03b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3770006fd0 tx=0x7f3770006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.092+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f377c080d20 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.368+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f378019cd40 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.368+0000 7f37777fe700 1 -- 172.21.15.121:0/2591325462 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f377c02d6f0 con 0x7f3780106540 2024-02-10T05:52:15.990 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 msgr2=0x7f376c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 0x7f376c03b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3770006fd0 tx=0x7f3770006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 msgr2=0x7f3780199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f377c003da0 tx=0x7f377c011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 shutdown_connections 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f376c038e40 0x7f376c03b300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 --2- 172.21.15.121:0/2591325462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3780106540 0x7f3780199700 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 >> 172.21.15.121:0/2591325462 conn(0x7f37800fffd0 msgr2=0x7f3780100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 shutdown_connections 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:10.370+0000 7f37887f6700 1 -- 172.21.15.121:0/2591325462 wait complete. 2024-02-10T05:52:15.991 INFO:teuthology.orchestra.run.smithi121.stdout:mgr is available 2024-02-10T05:52:16.906 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2020515221' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:52:17.208 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-02-10T05:52:17.209 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.403+0000 7fdc3dbd9700 1 Processor -- start 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc3dbd9700 1 -- start start 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc3dbd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc3dbd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38105390 con 0x7fdc381049a0 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58390/0 (socket says 172.21.15.121:58390) 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc377fe700 1 -- 172.21.15.121:0/2844466998 learned_addr learned my addr 172.21.15.121:0/2844466998 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc377fe700 1 -- 172.21.15.121:0/2844466998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38105ba0 con 0x7fdc381049a0 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc377fe700 1 --2- 172.21.15.121:0/2844466998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0098a0 tx=0x7fdc2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bf98458c4edea1 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.404+0000 7fdc367fc700 1 -- 172.21.15.121:0/2844466998 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc2c004160 con 0x7fdc381049a0 2024-02-10T05:52:17.210 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc367fc700 1 -- 172.21.15.121:0/2844466998 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fdc2c0042c0 con 0x7fdc381049a0 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc367fc700 1 -- 172.21.15.121:0/2844466998 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc2c0118a0 con 0x7fdc381049a0 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2844466998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 msgr2=0x7fdc38104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2844466998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c0098a0 tx=0x7fdc2c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2844466998 shutdown_connections 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2844466998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38104dc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2844466998 >> 172.21.15.121:0/2844466998 conn(0x7fdc380fffa0 msgr2=0x7fdc381023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2844466998 shutdown_connections 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.405+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2844466998 wait complete. 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc3dbd9700 1 Processor -- start 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc3dbd9700 1 -- start start 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc3dbd9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc3dbd9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc38198bd0 con 0x7fdc381049a0 2024-02-10T05:52:17.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58392/0 (socket says 172.21.15.121:58392) 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc377fe700 1 -- 172.21.15.121:0/2020515221 learned_addr learned my addr 172.21.15.121:0/2020515221 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.406+0000 7fdc377fe700 1 -- 172.21.15.121:0/2020515221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c009550 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc377fe700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c003f10 tx=0x7fdc2c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc2c0036a0 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fdc2c01a430 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc38198dd0 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdc2c024410 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.407+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc38199270 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45921+0+0 (secure 0 0 0) 0x7fdc2c02d3f0 con 0x7fdc381049a0 2024-02-10T05:52:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc34ff9700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 0x7fdc2003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc38063350 con 0x7fdc381049a0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdc2c04c890 con 0x7fdc381049a0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc36ffd700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 0x7fdc2003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.408+0000 7fdc36ffd700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 0x7fdc2003b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc28006fd0 tx=0x7fdc28006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.413+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc2c010970 con 0x7fdc381049a0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.605+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdc38108dd0 con 0x7fdc381049a0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.606+0000 7fdc34ff9700 1 -- 172.21.15.121:0/2020515221 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7fdc2c010380 con 0x7fdc381049a0 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 msgr2=0x7fdc2003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 0x7fdc2003b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdc28006fd0 tx=0x7fdc28006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 msgr2=0x7fdc38198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c003f10 tx=0x7fdc2c011940 comp rx=0 tx=0).stop 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 shutdown_connections 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fdc20038ee0 0x7fdc2003b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:17.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 --2- 172.21.15.121:0/2020515221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc381049a0 0x7fdc38198690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:17.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 >> 172.21.15.121:0/2020515221 conn(0x7fdc380fffa0 msgr2=0x7fdc38100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:17.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 shutdown_connections 2024-02-10T05:52:17.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:16.608+0000 7fdc3dbd9700 1 -- 172.21.15.121:0/2020515221 wait complete. 2024-02-10T05:52:17.214 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling cephadm module... 2024-02-10T05:52:19.188 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3085876333' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T05:52:19.188 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:18 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setuser ceph since I am not root 2024-02-10T05:52:19.188 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:18 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setgroup ceph since I am not root 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.634+0000 7ff90f014700 1 Processor -- start 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.634+0000 7ff90f014700 1 -- start start 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90f014700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90f014700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9081053a0 con 0x7ff9081049b0 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90cdb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90cdb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57038/0 (socket says 172.21.15.121:57038) 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90cdb0700 1 -- 172.21.15.121:0/3863353159 learned_addr learned my addr 172.21.15.121:0/3863353159 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90cdb0700 1 -- 172.21.15.121:0/3863353159 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff908105bb0 con 0x7ff9081049b0 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff90cdb0700 1 --2- 172.21.15.121:0/3863353159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0098a0 tx=0x7ff8fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c77d3154ba7c60a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:19.480 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff9077fe700 1 -- 172.21.15.121:0/3863353159 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8fc004160 con 0x7ff9081049b0 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff9077fe700 1 -- 172.21.15.121:0/3863353159 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7ff8fc0042c0 con 0x7ff9081049b0 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.635+0000 7ff9077fe700 1 -- 172.21.15.121:0/3863353159 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8fc0118a0 con 0x7ff9081049b0 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 -- 172.21.15.121:0/3863353159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 msgr2=0x7ff908104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 --2- 172.21.15.121:0/3863353159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0098a0 tx=0x7ff8fc009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 -- 172.21.15.121:0/3863353159 shutdown_connections 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 --2- 172.21.15.121:0/3863353159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff908104dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 -- 172.21.15.121:0/3863353159 >> 172.21.15.121:0/3863353159 conn(0x7ff9080fff70 msgr2=0x7ff908102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 -- 172.21.15.121:0/3863353159 shutdown_connections 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.636+0000 7ff90f014700 1 -- 172.21.15.121:0/3863353159 wait complete. 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90f014700 1 Processor -- start 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90f014700 1 -- start start 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90f014700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90f014700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90818fee0 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90cdb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90cdb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57054/0 (socket says 172.21.15.121:57054) 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90cdb0700 1 -- 172.21.15.121:0/3085876333 learned_addr learned my addr 172.21.15.121:0/3085876333 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90cdb0700 1 -- 172.21.15.121:0/3085876333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc009550 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff90cdb0700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc003da0 tx=0x7ff8fc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.637+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8fc011930 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.638+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90818e0d0 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.638+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7ff8fc003b00 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.638+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8fc01ad50 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.638+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90818e570 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.638+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 45921+0+0 (secure 0 0 0) 0x7ff8fc02d440 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.639+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90818ef80 con 0x7ff9081049b0 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.639+0000 7ff905ffb700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 0x7ff8f803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:19.482 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.639+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff8fc04ce30 con 0x7ff9081049b0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.639+0000 7ff907fff700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 0x7ff8f803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.639+0000 7ff907fff700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 0x7ff8f803b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006fd0 tx=0x7ff8f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.644+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff8fc02d6f0 con 0x7ff9081049b0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:17.890+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff908063350 con 0x7ff9081049b0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.873+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff8fc018860 con 0x7ff9081049b0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.875+0000 7ff905ffb700 1 -- 172.21.15.121:0/3085876333 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7ff8fc0194b0 con 0x7ff9081049b0 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.876+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 msgr2=0x7ff8f803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.876+0000 7ff90f014700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 0x7ff8f803b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006fd0 tx=0x7ff8f4006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.876+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 msgr2=0x7ff90818f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.876+0000 7ff90f014700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc003da0 tx=0x7ff8fc011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 shutdown_connections 2024-02-10T05:52:19.483 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7ff8f8038e40 0x7ff8f803b300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:19.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 --2- 172.21.15.121:0/3085876333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9081049b0 0x7ff90818f9a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:19.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 >> 172.21.15.121:0/3085876333 conn(0x7ff9080fff70 msgr2=0x7ff908100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:19.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 shutdown_connections 2024-02-10T05:52:19.484 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:18.877+0000 7ff90f014700 1 -- 172.21.15.121:0/3085876333 wait complete. 2024-02-10T05:52:19.726 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:19.450+0000 7f8f80967200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:52:19.726 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:19.557+0000 7f8f80967200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:52:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3085876333' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T05:52:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:19 smithi121 ceph-mon[30020]: mgrmap e5: y(active, since 12s) 2024-02-10T05:52:20.469 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.232+0000 7f8f80967200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:52:20.470 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.326+0000 7f8f80967200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:52:20.470 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.468+0000 7f8f80967200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:52:20.762 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.565+0000 7f8f80967200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:52:20.762 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.648+0000 7f8f80967200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:52:20.770 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.943+0000 7fd772a92700 1 Processor -- start 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.943+0000 7fd772a92700 1 -- start start 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.943+0000 7fd772a92700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.943+0000 7fd772a92700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c105880 con 0x7fd76c104e90 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.943+0000 7fd77082e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd77082e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57078/0 (socket says 172.21.15.121:57078) 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd77082e700 1 -- 172.21.15.121:0/789953421 learned_addr learned my addr 172.21.15.121:0/789953421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd77082e700 1 -- 172.21.15.121:0/789953421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c105bb0 con 0x7fd76c104e90 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd77082e700 1 --2- 172.21.15.121:0/789953421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7600098a0 tx=0x7fd760009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509de7511b04a9b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd76b7fe700 1 -- 172.21.15.121:0/789953421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd760004160 con 0x7fd76c104e90 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd76b7fe700 1 -- 172.21.15.121:0/789953421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd7600042c0 con 0x7fd76c104e90 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.944+0000 7fd76b7fe700 1 -- 172.21.15.121:0/789953421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7600118a0 con 0x7fd76c104e90 2024-02-10T05:52:20.771 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 -- 172.21.15.121:0/789953421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 msgr2=0x7fd76c1052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 --2- 172.21.15.121:0/789953421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7600098a0 tx=0x7fd760009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 -- 172.21.15.121:0/789953421 shutdown_connections 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 --2- 172.21.15.121:0/789953421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c1052b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 -- 172.21.15.121:0/789953421 >> 172.21.15.121:0/789953421 conn(0x7fd76c0ff750 msgr2=0x7fd76c101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 -- 172.21.15.121:0/789953421 shutdown_connections 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 -- 172.21.15.121:0/789953421 wait complete. 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.945+0000 7fd772a92700 1 Processor -- start 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd772a92700 1 -- start start 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd772a92700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd772a92700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1049b0 con 0x7fd76c104e90 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd77082e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:20.772 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd77082e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57090/0 (socket says 172.21.15.121:57090) 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd77082e700 1 -- 172.21.15.121:0/3613276712 learned_addr learned my addr 172.21.15.121:0/3613276712 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd77082e700 1 -- 172.21.15.121:0/3613276712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd760009550 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd77082e700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd760011890 tx=0x7fd760011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7600036a0 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd76001a430 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c19c4a0 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.946+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd7600193c0 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.947+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c19c940 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.947+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fd7600195f0 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.948+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c063350 con 0x7fd76c104e90 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.948+0000 7fd769ffb700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 0x7fd75c03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.948+0000 7fd76bfff700 1 -- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 msgr2=0x7fd75c03b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:20.773 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.948+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd76004cfe0 con 0x7fd76c104e90 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.948+0000 7fd76bfff700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 0x7fd75c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:19.953+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd760018940 con 0x7fd76c104e90 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.149+0000 7fd76bfff700 1 -- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 msgr2=0x7fd75c03b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.149+0000 7fd76bfff700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 0x7fd75c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.210+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd76c19ce00 con 0x7fd76c104e90 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.211+0000 7fd769ffb700 1 -- 172.21.15.121:0/3613276712 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd76004ca60 con 0x7fd76c104e90 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 msgr2=0x7fd75c03b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 0x7fd75c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 msgr2=0x7fd76c104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd760011890 tx=0x7fd760011970 comp rx=0 tx=0).stop 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 shutdown_connections 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7fd75c038f30 0x7fd75c03b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 --2- 172.21.15.121:0/3613276712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd76c104e90 0x7fd76c104470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:20.774 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 >> 172.21.15.121:0/3613276712 conn(0x7fd76c0ff750 msgr2=0x7fd76c100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:20.775 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 shutdown_connections 2024-02-10T05:52:20.775 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:20.214+0000 7fd772a92700 1 -- 172.21.15.121:0/3613276712 wait complete. 2024-02-10T05:52:20.775 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-02-10T05:52:20.775 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 5... 2024-02-10T05:52:21.033 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3613276712' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:52:21.033 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:20 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:20.914+0000 7f8f80967200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:52:21.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:21 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:21.032+0000 7f8f80967200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:21.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:21 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:21.118+0000 7f8f80967200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:52:21.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:21 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:21.217+0000 7f8f80967200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:52:21.828 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:21 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:21.394+0000 7f8f80967200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:52:22.632 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:22.343+0000 7f8f80967200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:52:22.632 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:22.471+0000 7f8f80967200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:52:22.890 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:22.800+0000 7f8f80967200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:52:22.891 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:22.890+0000 7f8f80967200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:52:23.267 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:23 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:23.083+0000 7f8f80967200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:23.267 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:23 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:23.263+0000 7f8f80967200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:52:23.799 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:23 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:23.543+0000 7f8f80967200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:23.799 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:23 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:23.798+0000 7f8f80967200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:24.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:24 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:24.030+0000 7f8f80967200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:52:25.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:24 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:24.818+0000 7f8f80967200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:52:25.690 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:25.380+0000 7f8f80967200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:52:25.690 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:25.477+0000 7f8f80967200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:52:25.690 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:25.689+0000 7f8f80967200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: Active manager daemon y restarted 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: Activating manager daemon y 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: mgrmap e6: y(active, starting, since 0.00329555s) 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: Manager daemon y is now available 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:25 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:27.016 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: Found migration_current of "None". Setting to last migration. 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:27.017 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:26 smithi121 ceph-mon[30020]: mgrmap e7: y(active, since 1.00554s) 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18e5931700 1 Processor -- start 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18e5931700 1 -- start start 2024-02-10T05:52:27.218 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18e5931700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18e5931700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e01069a0 con 0x7f18e0105fb0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57106/0 (socket says 172.21.15.121:57106) 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18deffd700 1 -- 172.21.15.121:0/406290569 learned_addr learned my addr 172.21.15.121:0/406290569 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18deffd700 1 -- 172.21.15.121:0/406290569 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0106ae0 con 0x7f18e0105fb0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.225+0000 7f18deffd700 1 --2- 172.21.15.121:0/406290569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f18d00098a0 tx=0x7f18d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d624b24eade598c server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18ddffb700 1 -- 172.21.15.121:0/406290569 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d0004160 con 0x7f18e0105fb0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18ddffb700 1 -- 172.21.15.121:0/406290569 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f18d00042c0 con 0x7f18e0105fb0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18ddffb700 1 -- 172.21.15.121:0/406290569 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d00118a0 con 0x7f18e0105fb0 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 -- 172.21.15.121:0/406290569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 msgr2=0x7f18e01063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 --2- 172.21.15.121:0/406290569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f18d00098a0 tx=0x7f18d0009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 -- 172.21.15.121:0/406290569 shutdown_connections 2024-02-10T05:52:27.219 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 --2- 172.21.15.121:0/406290569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e01063d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 -- 172.21.15.121:0/406290569 >> 172.21.15.121:0/406290569 conn(0x7f18e00ff780 msgr2=0x7f18e0101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 -- 172.21.15.121:0/406290569 shutdown_connections 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.226+0000 7f18e5931700 1 -- 172.21.15.121:0/406290569 wait complete. 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.227+0000 7f18e5931700 1 Processor -- start 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.227+0000 7f18e5931700 1 -- start start 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.227+0000 7f18e5931700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.227+0000 7f18e5931700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e019be80 con 0x7f18e0105fb0 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57116/0 (socket says 172.21.15.121:57116) 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18deffd700 1 -- 172.21.15.121:0/3207482607 learned_addr learned my addr 172.21.15.121:0/3207482607 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18deffd700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d0009550 con 0x7f18e0105fb0 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18deffd700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18d0003da0 tx=0x7f18d0011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:27.220 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d0011930 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e019c080 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f18d0003b00 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18d001ad50 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.228+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e019c520 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f18d002d440 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18e492f700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18e0104690 con 0x7f18c8038e90 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f18d004ce70 con 0x7f18e0105fb0 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18de7fc700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.229+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.430+0000 7f18de7fc700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.430+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.831+0000 7f18de7fc700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:27.221 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:21.831+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:22.632+0000 7f18de7fc700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:22.632+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:24.233+0000 7f18de7fc700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2998711588 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:24.233+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:25.705+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 6) v1 ==== 45699+0+0 (secure 0 0 0) 0x7f18d001a630 con 0x7f18e0105fb0 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:25.705+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 msgr2=0x7f18c803b350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:25.705+0000 7f18e492f700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.706+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 7) v1 ==== 45826+0+0 (secure 0 0 0) 0x7f18d004e340 con 0x7f18e0105fb0 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.706+0000 7f18e492f700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 0x7f18c803e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.706+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18e0104690 con 0x7f18c803c5d0 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.707+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 0x7f18c803e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.707+0000 7f18de7fc700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 0x7f18c803e9c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f18d4003a10 tx=0x7f18d4009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:27.222 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.708+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f18e0104690 con 0x7f18c803c5d0 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.714+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f18e0104800 con 0x7f18c803c5d0 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e492f700 1 -- 172.21.15.121:0/3207482607 <== mgr.14118 v2:172.21.15.121:6800/2724527041 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f18e0104800 con 0x7f18c803c5d0 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 msgr2=0x7f18c803e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 0x7f18c803e9c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f18d4003a10 tx=0x7f18d4009290 comp rx=0 tx=0).stop 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 msgr2=0x7f18e019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18d0003da0 tx=0x7f18d0011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 shutdown_connections 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18e0105fb0 0x7f18e019b940 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f18c803c5d0 0x7f18c803e9c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 --2- 172.21.15.121:0/3207482607 >> [v2:172.21.15.121:6800/2998711588,v1:172.21.15.121:6801/2998711588] conn(0x7f18c8038e90 0x7f18c803b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 >> 172.21.15.121:0/3207482607 conn(0x7f18e00ff780 msgr2=0x7f18e01920f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 shutdown_connections 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:26.715+0000 7f18e5931700 1 -- 172.21.15.121:0/3207482607 wait complete. 2024-02-10T05:52:27.223 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 5 is available 2024-02-10T05:52:27.224 INFO:teuthology.orchestra.run.smithi121.stdout:Setting orchestrator backend to cephadm... 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:26] ENGINE Bus STARTING 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:26] ENGINE Serving on https://172.21.15.121:7150 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:26] ENGINE Serving on http://172.21.15.121:8765 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:26] ENGINE Bus STARTED 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:52:27.921 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:27 smithi121 ceph-mon[30020]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:52:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.697+0000 7f19d00ea700 1 Processor -- start 2024-02-10T05:52:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.697+0000 7f19d00ea700 1 -- start start 2024-02-10T05:52:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.697+0000 7f19d00ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:28.431 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.697+0000 7f19d00ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8106480 con 0x7f19c8105a90 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cde86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cde86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45068/0 (socket says 172.21.15.121:45068) 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cde86700 1 -- 172.21.15.121:0/1431984878 learned_addr learned my addr 172.21.15.121:0/1431984878 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cde86700 1 -- 172.21.15.121:0/1431984878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8106c90 con 0x7f19c8105a90 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cde86700 1 --2- 172.21.15.121:0/1431984878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19c40098a0 tx=0x7f19c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5194fdba52e921b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cce84700 1 -- 172.21.15.121:0/1431984878 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c4004160 con 0x7f19c8105a90 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cce84700 1 -- 172.21.15.121:0/1431984878 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f19c40042c0 con 0x7f19c8105a90 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.698+0000 7f19cce84700 1 -- 172.21.15.121:0/1431984878 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c40118a0 con 0x7f19c8105a90 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- 172.21.15.121:0/1431984878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 msgr2=0x7f19c8105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1431984878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19c40098a0 tx=0x7f19c4009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- 172.21.15.121:0/1431984878 shutdown_connections 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1431984878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c8105eb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- 172.21.15.121:0/1431984878 >> 172.21.15.121:0/1431984878 conn(0x7f19c8101070 msgr2=0x7f19c8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:28.432 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- 172.21.15.121:0/1431984878 shutdown_connections 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- 172.21.15.121:0/1431984878 wait complete. 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 Processor -- start 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.699+0000 7f19d00ea700 1 -- start start 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19d00ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19d00ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c8199c30 con 0x7f19c8105a90 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19cde86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19cde86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45070/0 (socket says 172.21.15.121:45070) 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19cde86700 1 -- 172.21.15.121:0/1306517691 learned_addr learned my addr 172.21.15.121:0/1306517691 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19cde86700 1 -- 172.21.15.121:0/1306517691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4009550 con 0x7f19c8105a90 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19cde86700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19c4003da0 tx=0x7f19c4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c4011930 con 0x7f19c8105a90 2024-02-10T05:52:28.433 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f19c4003b00 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f19c401ad50 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c8199e30 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.700+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c819a2d0 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.701+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 7) v1 ==== 45826+0+0 (secure 0 0 0) 0x7f19c401a630 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.701+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c8063350 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.701+0000 7f19beffd700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 0x7f19b403b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.701+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f19c404cc40 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.701+0000 7f19cd685700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 0x7f19b403b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.702+0000 7f19cd685700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 0x7f19b403b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006fd0 tx=0x7f19b8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.706+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f19c40036a0 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.757+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f19c402d960 con 0x7f19c8105a90 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.920+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f19c8102270 con 0x7f19b4038dd0 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.927+0000 7f19beffd700 1 -- 172.21.15.121:0/1306517691 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19c8102270 con 0x7f19b4038dd0 2024-02-10T05:52:28.434 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.929+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 msgr2=0x7f19b403b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.929+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 0x7f19b403b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f19b8006fd0 tx=0x7f19b8006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.929+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 msgr2=0x7f19c81996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.929+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f19c4003da0 tx=0x7f19c4011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 shutdown_connections 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f19b4038dd0 0x7f19b403b290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 --2- 172.21.15.121:0/1306517691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19c8105a90 0x7f19c81996f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 >> 172.21.15.121:0/1306517691 conn(0x7f19c8101070 msgr2=0x7f19c8101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 shutdown_connections 2024-02-10T05:52:28.435 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:27.930+0000 7f19d00ea700 1 -- 172.21.15.121:0/1306517691 wait complete. 2024-02-10T05:52:29.070 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:28 smithi121 ceph-mon[30020]: mgrmap e8: y(active, since 2s) 2024-02-10T05:52:29.070 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:28 smithi121 ceph-mon[30020]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:29.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:28 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:29.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:28 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:29.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f0618552700 1 Processor -- start 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f0618552700 1 -- start start 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f0618552700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f0618552700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100796e0 con 0x7f0610106560 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f06162ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f06162ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45076/0 (socket says 172.21.15.121:45076) 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.868+0000 7f06162ee700 1 -- 172.21.15.121:0/987209670 learned_addr learned my addr 172.21.15.121:0/987209670 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f06162ee700 1 -- 172.21.15.121:0/987209670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0610108e90 con 0x7f0610106560 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f06162ee700 1 --2- 172.21.15.121:0/987209670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466a4647bcb035fe server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f06152ec700 1 -- 172.21.15.121:0/987209670 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f060c004160 con 0x7f0610106560 2024-02-10T05:52:29.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f06152ec700 1 -- 172.21.15.121:0/987209670 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f060c0042c0 con 0x7f0610106560 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f06152ec700 1 -- 172.21.15.121:0/987209670 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f060c0118a0 con 0x7f0610106560 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f0618552700 1 -- 172.21.15.121:0/987209670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 msgr2=0x7f0610108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.869+0000 7f0618552700 1 --2- 172.21.15.121:0/987209670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f060c0098a0 tx=0x7f060c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- 172.21.15.121:0/987209670 shutdown_connections 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 --2- 172.21.15.121:0/987209670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610108950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- 172.21.15.121:0/987209670 >> 172.21.15.121:0/987209670 conn(0x7f06100fff70 msgr2=0x7f0610102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- 172.21.15.121:0/987209670 shutdown_connections 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- 172.21.15.121:0/987209670 wait complete. 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 Processor -- start 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- start start 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:29.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f0618552700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0610199c40 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f06162ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f06162ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45088/0 (socket says 172.21.15.121:45088) 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.870+0000 7f06162ee700 1 -- 172.21.15.121:0/4239035080 learned_addr learned my addr 172.21.15.121:0/4239035080 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f06162ee700 1 -- 172.21.15.121:0/4239035080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c009550 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f06162ee700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f060c011890 tx=0x7f060c011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f060c0036a0 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0610199e40 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f060c01a430 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f060c0193c0 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.871+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061019a2e0 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f060c019640 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610109b70 con 0x7f0610106560 2024-02-10T05:52:29.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f06077fe700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 0x7f05fc03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f060c04d0d0 con 0x7f0610106560 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f0615aed700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 0x7f05fc03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.872+0000 7f0615aed700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 0x7f05fc03b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0600006fd0 tx=0x7f0600006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:28.877+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f060c02dba0 con 0x7f0610106560 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.070+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0610101100 con 0x7f05fc038ee0 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.070+0000 7f06077fe700 1 -- 172.21.15.121:0/4239035080 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0610101100 con 0x7f05fc038ee0 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 msgr2=0x7f05fc03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 0x7f05fc03b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0600006fd0 tx=0x7f0600006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 msgr2=0x7f0610199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f060c011890 tx=0x7f060c011970 comp rx=0 tx=0).stop 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 shutdown_connections 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f05fc038ee0 0x7f05fc03b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:29.658 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 --2- 172.21.15.121:0/4239035080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0610106560 0x7f0610199700 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.072+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 >> 172.21.15.121:0/4239035080 conn(0x7f06100fff70 msgr2=0x7f06101009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.073+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 shutdown_connections 2024-02-10T05:52:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:29.073+0000 7f0618552700 1 -- 172.21.15.121:0/4239035080 wait complete. 2024-02-10T05:52:29.659 INFO:teuthology.orchestra.run.smithi121.stdout:Generating ssh key... 2024-02-10T05:52:30.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:29 smithi121 ceph-mon[30020]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:30.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: Generating public/private rsa key pair. 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: Your identification has been saved in /tmp/tmp2cfchqg6/key. 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: Your public key has been saved in /tmp/tmp2cfchqg6/key.pub. 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: The key fingerprint is: 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: SHA256:18M7YrIpE6Iulo0KoQRxpUIVGyxO8PodtM7O05wFr04 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: The key's randomart image is: 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: +---[RSA 3072]----+ 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |+.+=o | 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |.* oo | 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |= +.. | 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |.+ . .. o | 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |o. o oS . + | 2024-02-10T05:52:30.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |oo +... o. o | 2024-02-10T05:52:30.580 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |o =.++E=. o o | 2024-02-10T05:52:30.580 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |o+.+..B = . . | 2024-02-10T05:52:30.580 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: |+o. oo.oo | 2024-02-10T05:52:30.580 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:30 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: +----[SHA256]-----+ 2024-02-10T05:52:31.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.098+0000 7f6246284700 1 Processor -- start 2024-02-10T05:52:31.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.098+0000 7f6246284700 1 -- start start 2024-02-10T05:52:31.086 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.098+0000 7f6246284700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.098+0000 7f6246284700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62401053a0 con 0x7f62401049b0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.098+0000 7f623ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45096/0 (socket says 172.21.15.121:45096) 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623ffff700 1 -- 172.21.15.121:0/3101175407 learned_addr learned my addr 172.21.15.121:0/3101175407 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623ffff700 1 -- 172.21.15.121:0/3101175407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240105bb0 con 0x7f62401049b0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623ffff700 1 --2- 172.21.15.121:0/3101175407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62300098a0 tx=0x7f6230009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e6417660109ee5 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623effd700 1 -- 172.21.15.121:0/3101175407 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6230004160 con 0x7f62401049b0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623effd700 1 -- 172.21.15.121:0/3101175407 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f62300042c0 con 0x7f62401049b0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.099+0000 7f623effd700 1 -- 172.21.15.121:0/3101175407 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62300118a0 con 0x7f62401049b0 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- 172.21.15.121:0/3101175407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 msgr2=0x7f6240104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 --2- 172.21.15.121:0/3101175407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62300098a0 tx=0x7f6230009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:31.087 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- 172.21.15.121:0/3101175407 shutdown_connections 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 --2- 172.21.15.121:0/3101175407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240104dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- 172.21.15.121:0/3101175407 >> 172.21.15.121:0/3101175407 conn(0x7f62400fff70 msgr2=0x7f6240102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- 172.21.15.121:0/3101175407 shutdown_connections 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- 172.21.15.121:0/3101175407 wait complete. 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 Processor -- start 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.100+0000 7f6246284700 1 -- start start 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f6246284700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f6246284700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6240197a40 con 0x7f62401049b0 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45110/0 (socket says 172.21.15.121:45110) 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623ffff700 1 -- 172.21.15.121:0/2840546065 learned_addr learned my addr 172.21.15.121:0/2840546065 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:31.088 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623ffff700 1 -- 172.21.15.121:0/2840546065 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6230009550 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623ffff700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6230003da0 tx=0x7f6230011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6230011930 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f6230003b00 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6240197c40 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f623001ad50 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.101+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62401980e0 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.102+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f623002d440 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.102+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6240063350 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.102+0000 7f623d7fa700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 0x7f622803d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.102+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f623004cfd0 con 0x7f62401049b0 2024-02-10T05:52:31.089 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.102+0000 7f623f7fe700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 0x7f622803d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.103+0000 7f623f7fe700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 0x7f622803d560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6234006fd0 tx=0x7f6234006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.107+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6230018a40 con 0x7f62401049b0 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.297+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f624018f430 con 0x7f622803b0a0 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.524+0000 7f623d7fa700 1 -- 172.21.15.121:0/2840546065 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f624018f430 con 0x7f622803b0a0 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 msgr2=0x7f622803d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 0x7f622803d560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6234006fd0 tx=0x7f6234006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 msgr2=0x7f6240197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6230003da0 tx=0x7f6230011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 shutdown_connections 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f622803b0a0 0x7f622803d560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 --2- 172.21.15.121:0/2840546065 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62401049b0 0x7f6240197500 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 >> 172.21.15.121:0/2840546065 conn(0x7f62400fff70 msgr2=0x7f624018dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 shutdown_connections 2024-02-10T05:52:31.090 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:30.527+0000 7f6246284700 1 -- 172.21.15.121:0/2840546065 wait complete. 2024-02-10T05:52:31.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:31 smithi121 ceph-mon[30020]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:31.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:31 smithi121 ceph-mon[30020]: Generating ssh key... 2024-02-10T05:52:31.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:31 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:31.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:31 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:32.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaKIT4FgRpaqI5IS5n63XmrLL5oVkAPmRzZbspKpqvVZOFgiyTXmWQrEWQ/ZyRt3lWHvoOIoFEY1ME8eXlxEUzCGf3vzYDCGKUHSyDBKzlQmN151Iz0YVuFhvrycp1voQCBbH1vvxm1gSZjq3bT+p/AC2zzV32p4ybTDKVOmCNN0aKMeVVl+V+D+9egbzaEaz852d1FPIvW64cPVGApbdyd5Qn2bN/NNkQgkJBuGfI2UnF71Siuarm/ulT9vdhjraikBhVhC1OkqHukerprXbzinPkiXZ67Pbhj+9ILTwikYgzgErsb8Gns5Yh0Aw5SMDMfKKPRzWVVMgsQv4xOHE9/3cdta/yrgres7X+NYeEz3iEpTlp5WHnwXAHHm2qaa9gjhF94kFlSe6YV143i2GPYrSWC9vvImpQuTZ9cw+QltlOvysO7pbbkVBHe4bwSDC9JrTFomxNdBC/UEkgaO9r8I24Gxy39OVnicVHjir3P6jFWNnqqSTBJDDDcrslYT0= ceph-616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:52:32.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba8dc91700 1 Processor -- start 2024-02-10T05:52:32.325 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba8dc91700 1 -- start start 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba8dc91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba8dc91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881053a0 con 0x7fba881049b0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45126/0 (socket says 172.21.15.121:45126) 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba877fe700 1 -- 172.21.15.121:0/1053185918 learned_addr learned my addr 172.21.15.121:0/1053185918 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.555+0000 7fba877fe700 1 -- 172.21.15.121:0/1053185918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88105bb0 con 0x7fba881049b0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba877fe700 1 --2- 172.21.15.121:0/1053185918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba780098a0 tx=0x7fba78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c608fb79d74b393e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba867fc700 1 -- 172.21.15.121:0/1053185918 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba78004160 con 0x7fba881049b0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba867fc700 1 -- 172.21.15.121:0/1053185918 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fba780042c0 con 0x7fba881049b0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba867fc700 1 -- 172.21.15.121:0/1053185918 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba780118a0 con 0x7fba881049b0 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 -- 172.21.15.121:0/1053185918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 msgr2=0x7fba88104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 --2- 172.21.15.121:0/1053185918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fba780098a0 tx=0x7fba78009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:32.326 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 -- 172.21.15.121:0/1053185918 shutdown_connections 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 --2- 172.21.15.121:0/1053185918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba88104dd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 -- 172.21.15.121:0/1053185918 >> 172.21.15.121:0/1053185918 conn(0x7fba880fff70 msgr2=0x7fba88102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 -- 172.21.15.121:0/1053185918 shutdown_connections 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.556+0000 7fba8dc91700 1 -- 172.21.15.121:0/1053185918 wait complete. 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba8dc91700 1 Processor -- start 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba8dc91700 1 -- start start 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba8dc91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba8dc91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba88079800 con 0x7fba881049b0 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45130/0 (socket says 172.21.15.121:45130) 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.557+0000 7fba877fe700 1 -- 172.21.15.121:0/2412988300 learned_addr learned my addr 172.21.15.121:0/2412988300 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:32.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba877fe700 1 -- 172.21.15.121:0/2412988300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78009550 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba877fe700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba78003f10 tx=0x7fba78011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba780036a0 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8819a250 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fba7801a430 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba78024410 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.558+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8819eb10 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fba7802d3f0 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba84ff9700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 0x7fba7003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fba7804cce0 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba88063350 con 0x7fba881049b0 2024-02-10T05:52:32.328 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba86ffd700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 0x7fba7003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.559+0000 7fba86ffd700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 0x7fba7003b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba7c006fd0 tx=0x7fba7c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.564+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba78010970 con 0x7fba881049b0 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.758+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fba88101170 con 0x7fba70038ee0 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.759+0000 7fba84ff9700 1 -- 172.21.15.121:0/2412988300 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fba88101170 con 0x7fba70038ee0 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 msgr2=0x7fba7003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 0x7fba7003b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba7c006fd0 tx=0x7fba7c006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 msgr2=0x7fba880792c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fba78003f10 tx=0x7fba78011940 comp rx=0 tx=0).stop 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 shutdown_connections 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fba70038ee0 0x7fba7003b3a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 --2- 172.21.15.121:0/2412988300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba881049b0 0x7fba880792c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 >> 172.21.15.121:0/2412988300 conn(0x7fba880fff70 msgr2=0x7fba88100a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 shutdown_connections 2024-02-10T05:52:32.329 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:31.762+0000 7fba8dc91700 1 -- 172.21.15.121:0/2412988300 wait complete. 2024-02-10T05:52:32.330 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:52:32.330 INFO:teuthology.orchestra.run.smithi121.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T05:52:32.330 INFO:teuthology.orchestra.run.smithi121.stdout:Adding host smithi121... 2024-02-10T05:52:32.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:32 smithi121 ceph-mon[30020]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:33 smithi121 ceph-mon[30020]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:33 smithi121 ceph-mon[30020]: Deploying cephadm binary to smithi121 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Added host 'smithi121' with addr '172.21.15.121' 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f75556df700 1 Processor -- start 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f75556df700 1 -- start start 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f75556df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f75556df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550105380 con 0x7f7550104990 2024-02-10T05:52:37.758 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f754effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f754effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45136/0 (socket says 172.21.15.121:45136) 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f754effd700 1 -- 172.21.15.121:0/3690285547 learned_addr learned my addr 172.21.15.121:0/3690285547 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.772+0000 7f754effd700 1 -- 172.21.15.121:0/3690285547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550105b90 con 0x7f7550104990 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f754effd700 1 --2- 172.21.15.121:0/3690285547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75400098a0 tx=0x7f7540009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c21acbbbd5f445 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f754dffb700 1 -- 172.21.15.121:0/3690285547 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7540004160 con 0x7f7550104990 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f754dffb700 1 -- 172.21.15.121:0/3690285547 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f75400042c0 con 0x7f7550104990 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f754dffb700 1 -- 172.21.15.121:0/3690285547 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75400118a0 con 0x7f7550104990 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 -- 172.21.15.121:0/3690285547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 msgr2=0x7f7550104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 --2- 172.21.15.121:0/3690285547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75400098a0 tx=0x7f7540009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 -- 172.21.15.121:0/3690285547 shutdown_connections 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 --2- 172.21.15.121:0/3690285547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550104db0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 -- 172.21.15.121:0/3690285547 >> 172.21.15.121:0/3690285547 conn(0x7f75500fffd0 msgr2=0x7f75501023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:37.759 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 -- 172.21.15.121:0/3690285547 shutdown_connections 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.773+0000 7f75556df700 1 -- 172.21.15.121:0/3690285547 wait complete. 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f75556df700 1 Processor -- start 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f75556df700 1 -- start start 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f75556df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f75556df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550078f70 con 0x7f7550104990 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f754effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f754effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45138/0 (socket says 172.21.15.121:45138) 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.774+0000 7f754effd700 1 -- 172.21.15.121:0/2622933933 learned_addr learned my addr 172.21.15.121:0/2622933933 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f754effd700 1 -- 172.21.15.121:0/2622933933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540009550 con 0x7f7550104990 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f754effd700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7540003f10 tx=0x7f7540011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:37.760 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75400036a0 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755019c4d0 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f754001a430 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7540024410 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.775+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755019c970 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f754002d3f0 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7550063350 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f7537fff700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 0x7f753803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f754004ccf0 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f754e7fc700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 0x7f753803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.776+0000 7f754e7fc700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 0x7f753803b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7544006fd0 tx=0x7f7544006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.781+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7540051070 con 0x7f7550104990 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:32.980+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}) v1 -- 0x7f7550108d90 con 0x7f7538038ee0 2024-02-10T05:52:37.761 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.163+0000 7f7537fff700 1 -- 172.21.15.121:0/2622933933 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f7550108d90 con 0x7f7538038ee0 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 msgr2=0x7f753803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 0x7f753803b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7544006fd0 tx=0x7f7544006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 msgr2=0x7f7550078a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7540003f10 tx=0x7f7540011940 comp rx=0 tx=0).stop 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 shutdown_connections 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7538038ee0 0x7f753803b3a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 --2- 172.21.15.121:0/2622933933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7550104990 0x7f7550078a30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.166+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 >> 172.21.15.121:0/2622933933 conn(0x7f75500fffd0 msgr2=0x7f7550100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.167+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 shutdown_connections 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:37.167+0000 7f75556df700 1 -- 172.21.15.121:0/2622933933 wait complete. 2024-02-10T05:52:37.762 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying unmanaged mon service... 2024-02-10T05:52:38.293 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:38 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:38.293 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:38 smithi121 ceph-mon[30020]: Added host smithi121 2024-02-10T05:52:38.293 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:38 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:39.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:39 smithi121 ceph-mon[30020]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:39.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:39 smithi121 ceph-mon[30020]: Saving service mon spec with placement count:5 2024-02-10T05:52:39.813 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:39 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:40.248 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T05:52:40.248 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f1342247700 1 Processor -- start 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f1342247700 1 -- start start 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f1342247700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f1342247700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c105390 con 0x7f133c1049a0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f133b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f133b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51748/0 (socket says 172.21.15.121:51748) 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.292+0000 7f133b7fe700 1 -- 172.21.15.121:0/3122950131 learned_addr learned my addr 172.21.15.121:0/3122950131 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.293+0000 7f133b7fe700 1 -- 172.21.15.121:0/3122950131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c105ba0 con 0x7f133c1049a0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.293+0000 7f133b7fe700 1 --2- 172.21.15.121:0/3122950131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f132c0098a0 tx=0x7f132c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695226df7ba54f62 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.293+0000 7f133a7fc700 1 -- 172.21.15.121:0/3122950131 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f132c004160 con 0x7f133c1049a0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.293+0000 7f133a7fc700 1 -- 172.21.15.121:0/3122950131 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f132c0042c0 con 0x7f133c1049a0 2024-02-10T05:52:40.249 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.293+0000 7f133a7fc700 1 -- 172.21.15.121:0/3122950131 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f132c0118a0 con 0x7f133c1049a0 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 -- 172.21.15.121:0/3122950131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 msgr2=0x7f133c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 --2- 172.21.15.121:0/3122950131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f132c0098a0 tx=0x7f132c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 -- 172.21.15.121:0/3122950131 shutdown_connections 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 --2- 172.21.15.121:0/3122950131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c104dc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 -- 172.21.15.121:0/3122950131 >> 172.21.15.121:0/3122950131 conn(0x7f133c0fff80 msgr2=0x7f133c1023a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 -- 172.21.15.121:0/3122950131 shutdown_connections 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 -- 172.21.15.121:0/3122950131 wait complete. 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.294+0000 7f1342247700 1 Processor -- start 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1342247700 1 -- start start 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1342247700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1342247700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c199c80 con 0x7f133c1049a0 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f133b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:40.250 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f133b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51752/0 (socket says 172.21.15.121:51752) 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f133b7fe700 1 -- 172.21.15.121:0/4007231039 learned_addr learned my addr 172.21.15.121:0/4007231039 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f133b7fe700 1 -- 172.21.15.121:0/4007231039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c009550 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f133b7fe700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f132c003da0 tx=0x7f132c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f132c011930 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f132c003b00 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c199e80 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f132c01ad50 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.295+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c19a320 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.296+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f132c0036a0 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.296+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c1938c0 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.296+0000 7f1338ff9700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 0x7f132403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.297+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f132c04ce20 con 0x7f133c1049a0 2024-02-10T05:52:40.251 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.297+0000 7f133affd700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 0x7f132403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.297+0000 7f133affd700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 0x7f132403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.301+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f132c01a630 con 0x7f133c1049a0 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.519+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f133c02d8d0 con 0x7f1324038ee0 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.526+0000 7f1338ff9700 1 -- 172.21.15.121:0/4007231039 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f133c02d8d0 con 0x7f1324038ee0 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 msgr2=0x7f132403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 0x7f132403b3a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 msgr2=0x7f133c199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f132c003da0 tx=0x7f132c011e90 comp rx=0 tx=0).stop 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 shutdown_connections 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f1324038ee0 0x7f132403b3a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 --2- 172.21.15.121:0/4007231039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f133c1049a0 0x7f133c199740 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 >> 172.21.15.121:0/4007231039 conn(0x7f133c0fff80 msgr2=0x7f133c100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:40.252 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.528+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 shutdown_connections 2024-02-10T05:52:40.253 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:38.529+0000 7f1342247700 1 -- 172.21.15.121:0/4007231039 wait complete. 2024-02-10T05:52:40.253 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying unmanaged mgr service... 2024-02-10T05:52:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:40 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:40 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:41.554 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T05:52:41.584 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f7944999700 1 Processor -- start 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f7944999700 1 -- start start 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f7944999700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f7944999700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940078e90 con 0x7f7940106500 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f793e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f793e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51754/0 (socket says 172.21.15.121:51754) 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.741+0000 7f793e59c700 1 -- 172.21.15.121:0/4281596845 learned_addr learned my addr 172.21.15.121:0/4281596845 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.742+0000 7f793e59c700 1 -- 172.21.15.121:0/4281596845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940108e30 con 0x7f7940106500 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.742+0000 7f793e59c700 1 --2- 172.21.15.121:0/4281596845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79300098a0 tx=0x7f7930009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da63225e5b41bb51 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.742+0000 7f793d59a700 1 -- 172.21.15.121:0/4281596845 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7930004160 con 0x7f7940106500 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.742+0000 7f793d59a700 1 -- 172.21.15.121:0/4281596845 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f79300042c0 con 0x7f7940106500 2024-02-10T05:52:41.585 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.742+0000 7f793d59a700 1 -- 172.21.15.121:0/4281596845 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7930011850 con 0x7f7940106500 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 -- 172.21.15.121:0/4281596845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 msgr2=0x7f79401088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 --2- 172.21.15.121:0/4281596845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79300098a0 tx=0x7f7930009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 -- 172.21.15.121:0/4281596845 shutdown_connections 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 --2- 172.21.15.121:0/4281596845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f79401088f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 -- 172.21.15.121:0/4281596845 >> 172.21.15.121:0/4281596845 conn(0x7f79400fffd0 msgr2=0x7f79401023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 -- 172.21.15.121:0/4281596845 shutdown_connections 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 -- 172.21.15.121:0/4281596845 wait complete. 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.743+0000 7f7944999700 1 Processor -- start 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f7944999700 1 -- start start 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f7944999700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f7944999700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7940199c70 con 0x7f7940106500 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f793e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:41.586 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f793e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51756/0 (socket says 172.21.15.121:51756) 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f793e59c700 1 -- 172.21.15.121:0/2713998854 learned_addr learned my addr 172.21.15.121:0/2713998854 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f793e59c700 1 -- 172.21.15.121:0/2713998854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930009550 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f793e59c700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7930006af0 tx=0x7f7930011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79300036a0 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.744+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f793001a430 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.745+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7940199e70 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.745+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79300193c0 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.745+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794019a310 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f7930019630 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7940109bb0 con 0x7f7940106500 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f792f7fe700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 0x7f792803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:41.587 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f793004cc60 con 0x7f7940106500 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f793dd9b700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 0x7f792803b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.746+0000 7f793dd9b700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 0x7f792803b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7934006fd0 tx=0x7f7934006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.751+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7930023c20 con 0x7f7940106500 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.962+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f79401011a0 con 0x7f7928038ee0 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.969+0000 7f792f7fe700 1 -- 172.21.15.121:0/2713998854 <== mgr.14118 v2:172.21.15.121:6800/2724527041 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f79401011a0 con 0x7f7928038ee0 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 msgr2=0x7f792803b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 0x7f792803b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7934006fd0 tx=0x7f7934006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 msgr2=0x7f7940199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7930006af0 tx=0x7f7930011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 shutdown_connections 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f7928038ee0 0x7f792803b3a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:41.588 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 --2- 172.21.15.121:0/2713998854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7940106500 0x7f7940199730 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:41.589 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.971+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 >> 172.21.15.121:0/2713998854 conn(0x7f79400fffd0 msgr2=0x7f7940100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:41.589 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.972+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 shutdown_connections 2024-02-10T05:52:41.589 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:40.972+0000 7f7944999700 1 -- 172.21.15.121:0/2713998854 wait complete. 2024-02-10T05:52:42.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:41 smithi121 ceph-mon[30020]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:42.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:41 smithi121 ceph-mon[30020]: Saving service mgr spec with placement count:2 2024-02-10T05:52:42.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:41 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:42.862 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.019+0000 7f99801a9700 1 Processor -- start 2024-02-10T05:52:42.862 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.019+0000 7f99801a9700 1 -- start start 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.019+0000 7f99801a9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.019+0000 7f99801a9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978107a80 con 0x7f9978107090 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997df45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997df45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51760/0 (socket says 172.21.15.121:51760) 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997df45700 1 -- 172.21.15.121:0/2982069432 learned_addr learned my addr 172.21.15.121:0/2982069432 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997df45700 1 -- 172.21.15.121:0/2982069432 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978107bc0 con 0x7f9978107090 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997df45700 1 --2- 172.21.15.121:0/2982069432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99740098a0 tx=0x7f9974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe3da039070e9c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997cf43700 1 -- 172.21.15.121:0/2982069432 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9974004160 con 0x7f9978107090 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997cf43700 1 -- 172.21.15.121:0/2982069432 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f99740042c0 con 0x7f9978107090 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.020+0000 7f997cf43700 1 -- 172.21.15.121:0/2982069432 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9974011850 con 0x7f9978107090 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- 172.21.15.121:0/2982069432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 msgr2=0x7f99781074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 --2- 172.21.15.121:0/2982069432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f99740098a0 tx=0x7f9974009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:42.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- 172.21.15.121:0/2982069432 shutdown_connections 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 --2- 172.21.15.121:0/2982069432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f99781074b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- 172.21.15.121:0/2982069432 >> 172.21.15.121:0/2982069432 conn(0x7f99780ff750 msgr2=0x7f9978101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- 172.21.15.121:0/2982069432 shutdown_connections 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- 172.21.15.121:0/2982069432 wait complete. 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 Processor -- start 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.021+0000 7f99801a9700 1 -- start start 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f99801a9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f99801a9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978199cc0 con 0x7f9978107090 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f997df45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f997df45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51762/0 (socket says 172.21.15.121:51762) 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f997df45700 1 -- 172.21.15.121:0/3142002840 learned_addr learned my addr 172.21.15.121:0/3142002840 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f997df45700 1 -- 172.21.15.121:0/3142002840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974009550 con 0x7f9978107090 2024-02-10T05:52:42.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f997df45700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9974003d50 tx=0x7f9974011910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99740036a0 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9978199ec0 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9974024430 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f997401a430 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.022+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997819a360 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.023+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99781046a0 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.023+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f997401a6f0 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.023+0000 7f996effd700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 0x7f996403b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.023+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f997404cf00 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.023+0000 7f997d744700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 0x7f996403b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.024+0000 7f997d744700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 0x7f996403b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9968006fd0 tx=0x7f9968006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.028+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f997401a9a0 con 0x7f9978107090 2024-02-10T05:52:42.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.224+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f997819cd40 con 0x7f9978107090 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.229+0000 7f996effd700 1 -- 172.21.15.121:0/3142002840 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9974018e60 con 0x7f9978107090 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 msgr2=0x7f996403b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 0x7f996403b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9968006fd0 tx=0x7f9968006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 msgr2=0x7f9978199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9974003d50 tx=0x7f9974011910 comp rx=0 tx=0).stop 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 shutdown_connections 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f9964038e90 0x7f996403b350 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 --2- 172.21.15.121:0/3142002840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9978107090 0x7f9978199780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 >> 172.21.15.121:0/3142002840 conn(0x7f99780ff750 msgr2=0x7f99781002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 shutdown_connections 2024-02-10T05:52:42.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:42.231+0000 7f99801a9700 1 -- 172.21.15.121:0/3142002840 wait complete. 2024-02-10T05:52:43.509 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3142002840' entity='client.admin' 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.615+0000 7f82b2bdc700 1 Processor -- start 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.615+0000 7f82b2bdc700 1 -- start start 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.615+0000 7f82b2bdc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.615+0000 7f82b2bdc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac105380 con 0x7f82ac104990 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82b0978700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82b0978700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51770/0 (socket says 172.21.15.121:51770) 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82b0978700 1 -- 172.21.15.121:0/3393868621 learned_addr learned my addr 172.21.15.121:0/3393868621 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82b0978700 1 -- 172.21.15.121:0/3393868621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac105b90 con 0x7f82ac104990 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82b0978700 1 --2- 172.21.15.121:0/3393868621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82a00098a0 tx=0x7f82a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45985e4b964d324f server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82ab7fe700 1 -- 172.21.15.121:0/3393868621 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a0004160 con 0x7f82ac104990 2024-02-10T05:52:44.446 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82ab7fe700 1 -- 172.21.15.121:0/3393868621 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f82a00042c0 con 0x7f82ac104990 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.616+0000 7f82ab7fe700 1 -- 172.21.15.121:0/3393868621 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a0011850 con 0x7f82ac104990 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 -- 172.21.15.121:0/3393868621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 msgr2=0x7f82ac104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/3393868621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82a00098a0 tx=0x7f82a0009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 -- 172.21.15.121:0/3393868621 shutdown_connections 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/3393868621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac104db0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 -- 172.21.15.121:0/3393868621 >> 172.21.15.121:0/3393868621 conn(0x7f82ac0fffd0 msgr2=0x7f82ac1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 -- 172.21.15.121:0/3393868621 shutdown_connections 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 -- 172.21.15.121:0/3393868621 wait complete. 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.617+0000 7f82b2bdc700 1 Processor -- start 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b2bdc700 1 -- start start 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b2bdc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b2bdc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac199cc0 con 0x7f82ac104990 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b0978700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:44.447 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b0978700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51778/0 (socket says 172.21.15.121:51778) 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b0978700 1 -- 172.21.15.121:0/1795074760 learned_addr learned my addr 172.21.15.121:0/1795074760 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b0978700 1 -- 172.21.15.121:0/1795074760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0009550 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b0978700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82a0006af0 tx=0x7f82a0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a00036a0 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f82a001a430 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac199ec0 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.618+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a00193c0 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.619+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac19a360 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.619+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f82a0019520 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.620+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac1938f0 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.620+0000 7f82a9ffb700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 0x7f829c03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.620+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f82a004cd90 con 0x7f82ac104990 2024-02-10T05:52:44.448 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.620+0000 7f82abfff700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 0x7f829c03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.620+0000 7f82abfff700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 0x7f829c03b3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8298006fd0 tx=0x7f8298006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.625+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82a0052020 con 0x7f82ac104990 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.821+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f82ac063350 con 0x7f82ac104990 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.824+0000 7f82a9ffb700 1 -- 172.21.15.121:0/1795074760 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f82a0023390 con 0x7f82ac104990 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 msgr2=0x7f829c03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 0x7f829c03b3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8298006fd0 tx=0x7f8298006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 msgr2=0x7f82ac199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f82a0006af0 tx=0x7f82a0011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 shutdown_connections 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f829c038ee0 0x7f829c03b3a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 --2- 172.21.15.121:0/1795074760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82ac104990 0x7f82ac199780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.826+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 >> 172.21.15.121:0/1795074760 conn(0x7f82ac0fffd0 msgr2=0x7f82ac100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.827+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 shutdown_connections 2024-02-10T05:52:44.449 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:43.827+0000 7f82b2bdc700 1 -- 172.21.15.121:0/1795074760 wait complete. 2024-02-10T05:52:44.450 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling the dashboard module... 2024-02-10T05:52:44.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:44 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:44.774 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1795074760' entity='client.admin' 2024-02-10T05:52:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:45 smithi121 ceph-mon[30020]: from='mgr.14118 172.21.15.121:0/4015051094' entity='mgr.y' 2024-02-10T05:52:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3305101532' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T05:52:46.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setuser ceph since I am not root 2024-02-10T05:52:46.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setgroup ceph since I am not root 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e2260b700 1 Processor -- start 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e2260b700 1 -- start start 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e2260b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e2260b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c105380 con 0x7f0e1c104990 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51792/0 (socket says 172.21.15.121:51792) 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e1bfff700 1 -- 172.21.15.121:0/3329804475 learned_addr learned my addr 172.21.15.121:0/3329804475 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.881+0000 7f0e1bfff700 1 -- 172.21.15.121:0/3329804475 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c105b90 con 0x7f0e1c104990 2024-02-10T05:52:46.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.882+0000 7f0e1bfff700 1 --2- 172.21.15.121:0/3329804475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0098a0 tx=0x7f0e0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51983018f69c0c85 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.882+0000 7f0e1affd700 1 -- 172.21.15.121:0/3329804475 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c004160 con 0x7f0e1c104990 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.882+0000 7f0e1affd700 1 -- 172.21.15.121:0/3329804475 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0e0c0042c0 con 0x7f0e1c104990 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.882+0000 7f0e1affd700 1 -- 172.21.15.121:0/3329804475 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c011850 con 0x7f0e1c104990 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 -- 172.21.15.121:0/3329804475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 msgr2=0x7f0e1c104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3329804475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0098a0 tx=0x7f0e0c009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 -- 172.21.15.121:0/3329804475 shutdown_connections 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3329804475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c104db0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 -- 172.21.15.121:0/3329804475 >> 172.21.15.121:0/3329804475 conn(0x7f0e1c0fffd0 msgr2=0x7f0e1c1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 -- 172.21.15.121:0/3329804475 shutdown_connections 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.883+0000 7f0e2260b700 1 -- 172.21.15.121:0/3329804475 wait complete. 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e2260b700 1 Processor -- start 2024-02-10T05:52:46.466 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e2260b700 1 -- start start 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e2260b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e2260b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c197ac0 con 0x7f0e1c104990 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e1bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51806/0 (socket says 172.21.15.121:51806) 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e1bfff700 1 -- 172.21.15.121:0/3305101532 learned_addr learned my addr 172.21.15.121:0/3305101532 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.884+0000 7f0e1bfff700 1 -- 172.21.15.121:0/3305101532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c009550 con 0x7f0e1c104990 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e1bfff700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006af0 tx=0x7f0e0c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c0036a0 con 0x7f0e1c104990 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c197cc0 con 0x7f0e1c104990 2024-02-10T05:52:46.467 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c198160 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0e0c01a430 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.885+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0e0c0193c0 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.886+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f0e0c019520 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.886+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08005150 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.886+0000 7f0e197fa700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 0x7f0e0403d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.886+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0e0c04dab0 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.886+0000 7f0e1b7fe700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 0x7f0e0403d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.887+0000 7f0e1b7fe700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 0x7f0e0403d5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006fd0 tx=0x7f0e10006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:44.895+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e0c02d430 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.149+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0e08005400 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.775+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0e0c023c20 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.777+0000 7f0e197fa700 1 -- 172.21.15.121:0/3305101532 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f0e0c02d7b0 con 0x7f0e1c104990 2024-02-10T05:52:46.468 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 msgr2=0x7f0e0403d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 0x7f0e0403d5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0e10006fd0 tx=0x7f0e10006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 msgr2=0x7f0e1c197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006af0 tx=0x7f0e0c011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 shutdown_connections 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f0e0403b110 0x7f0e0403d5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 --2- 172.21.15.121:0/3305101532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e1c104990 0x7f0e1c197580 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 >> 172.21.15.121:0/3305101532 conn(0x7f0e1c0fffd0 msgr2=0x7f0e1c18ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 shutdown_connections 2024-02-10T05:52:46.469 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:45.778+0000 7f0e2260b700 1 -- 172.21.15.121:0/3305101532 wait complete. 2024-02-10T05:52:46.698 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:46 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:46.345+0000 7f34f04d3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:52:46.698 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:46 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:46.440+0000 7f34f04d3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:52:47.069 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3305101532' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T05:52:47.069 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:46 smithi121 ceph-mon[30020]: mgrmap e9: y(active, since 20s) 2024-02-10T05:52:47.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.099+0000 7f34f04d3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:52:47.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.192+0000 7f34f04d3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:52:47.776 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.338+0000 7f34f04d3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:52:47.776 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.435+0000 7f34f04d3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:52:47.777 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.519+0000 7f34f04d3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:52:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1813701668' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:52:48.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.795+0000 7f34f04d3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:52:48.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:47 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:47.911+0000 7f34f04d3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:48.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:48.004+0000 7f34f04d3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.281+0000 7fb40a825700 1 Processor -- start 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.281+0000 7fb40a825700 1 -- start start 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.281+0000 7fb40a825700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:48.133 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.281+0000 7fb40a825700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404106480 con 0x7fb404105a90 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb403fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb403fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51842/0 (socket says 172.21.15.121:51842) 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb403fff700 1 -- 172.21.15.121:0/4246578355 learned_addr learned my addr 172.21.15.121:0/4246578355 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb403fff700 1 -- 172.21.15.121:0/4246578355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404106c90 con 0x7fb404105a90 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb403fff700 1 --2- 172.21.15.121:0/4246578355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098a0 tx=0x7fb3f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9f44bb9039862a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb402ffd700 1 -- 172.21.15.121:0/4246578355 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3f8004160 con 0x7fb404105a90 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb402ffd700 1 -- 172.21.15.121:0/4246578355 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb3f80042c0 con 0x7fb404105a90 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.282+0000 7fb402ffd700 1 -- 172.21.15.121:0/4246578355 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3f8011850 con 0x7fb404105a90 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 -- 172.21.15.121:0/4246578355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 msgr2=0x7fb404105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 --2- 172.21.15.121:0/4246578355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098a0 tx=0x7fb3f8009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 -- 172.21.15.121:0/4246578355 shutdown_connections 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 --2- 172.21.15.121:0/4246578355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404105eb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:48.134 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 -- 172.21.15.121:0/4246578355 >> 172.21.15.121:0/4246578355 conn(0x7fb404101090 msgr2=0x7fb4041034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 -- 172.21.15.121:0/4246578355 shutdown_connections 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.283+0000 7fb40a825700 1 -- 172.21.15.121:0/4246578355 wait complete. 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb40a825700 1 Processor -- start 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb40a825700 1 -- start start 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb40a825700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb40a825700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb404199c50 con 0x7fb404105a90 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb403fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb403fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51844/0 (socket says 172.21.15.121:51844) 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb403fff700 1 -- 172.21.15.121:0/1813701668 learned_addr learned my addr 172.21.15.121:0/1813701668 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.284+0000 7fb403fff700 1 -- 172.21.15.121:0/1813701668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8009550 con 0x7fb404105a90 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb403fff700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006af0 tx=0x7fb3f8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3f80036a0 con 0x7fb404105a90 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb3f801a430 con 0x7fb404105a90 2024-02-10T05:52:48.135 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb404199e50 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3f80193c0 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.285+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40419a2f0 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7fb3f8019630 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb4017fa700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 0x7fb3ec03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb3f804db40 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb4037fe700 1 -- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 msgr2=0x7fb3ec03b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb4037fe700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 0x7fb3ec03b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.286+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404063350 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.291+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3f80199d0 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.487+0000 7fb4037fe700 1 -- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 msgr2=0x7fb3ec03b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.487+0000 7fb4037fe700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 0x7fb3ec03b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.542+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb40419cbd0 con 0x7fb404105a90 2024-02-10T05:52:48.136 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.543+0000 7fb4017fa700 1 -- 172.21.15.121:0/1813701668 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb3f8023c20 con 0x7fb404105a90 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 msgr2=0x7fb3ec03b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 0x7fb3ec03b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 msgr2=0x7fb404199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006af0 tx=0x7fb3f8011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 shutdown_connections 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7fb3ec038ee0 0x7fb3ec03b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 --2- 172.21.15.121:0/1813701668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb404105a90 0x7fb404199710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 >> 172.21.15.121:0/1813701668 conn(0x7fb404101090 msgr2=0x7fb404101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 shutdown_connections 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:47.545+0000 7fb40a825700 1 -- 172.21.15.121:0/1813701668 wait complete. 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-02-10T05:52:48.137 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 9... 2024-02-10T05:52:48.360 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:48.104+0000 7f34f04d3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:52:48.360 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:48.290+0000 7f34f04d3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:52:49.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:49 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:49.272+0000 7f34f04d3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:52:49.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:49 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:49.401+0000 7f34f04d3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:52:50.019 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:49 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:49.732+0000 7f34f04d3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:52:50.019 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:49 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:49.824+0000 7f34f04d3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:52:50.019 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:50 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:50.017+0000 7f34f04d3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:50.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:50 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:50.201+0000 7f34f04d3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:52:50.741 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:50 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:50.484+0000 7f34f04d3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:52:50.741 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:50 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:50.740+0000 7f34f04d3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:52:51.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:50 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:50.973+0000 7f34f04d3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:52:52.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:51 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:51.773+0000 7f34f04d3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:52:52.659 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:52.341+0000 7f34f04d3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:52:52.659 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:52.440+0000 7f34f04d3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:52:52.659 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:52:52.657+0000 7f34f04d3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: Active manager daemon y restarted 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: Activating manager daemon y 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: mgrmap e10: y(active, starting, since 0.00332725s) 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:52:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:52:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: Manager daemon y is now available 2024-02-10T05:52:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:54.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:52:54.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:52:54.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:53 smithi121 ceph-mon[30020]: mgrmap e11: y(active, since 1.00559s) 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.624+0000 7f5e0ee62700 1 Processor -- start 2024-02-10T05:52:54.308 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.624+0000 7f5e0ee62700 1 -- start start 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.624+0000 7f5e0ee62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.624+0000 7f5e0ee62700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e08106480 con 0x7f5e08105a90 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e0cbfe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e0cbfe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45790/0 (socket says 172.21.15.121:45790) 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e0cbfe700 1 -- 172.21.15.121:0/3357587741 learned_addr learned my addr 172.21.15.121:0/3357587741 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e0cbfe700 1 -- 172.21.15.121:0/3357587741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08106c90 con 0x7f5e08105a90 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e0cbfe700 1 --2- 172.21.15.121:0/3357587741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098a0 tx=0x7f5dfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724d9779bef2f4c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e077fe700 1 -- 172.21.15.121:0/3357587741 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc004160 con 0x7f5e08105a90 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e077fe700 1 -- 172.21.15.121:0/3357587741 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5dfc0042c0 con 0x7f5e08105a90 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.625+0000 7f5e077fe700 1 -- 172.21.15.121:0/3357587741 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc011850 con 0x7f5e08105a90 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 -- 172.21.15.121:0/3357587741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 msgr2=0x7f5e08105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/3357587741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc0098a0 tx=0x7f5dfc009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.309 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 -- 172.21.15.121:0/3357587741 shutdown_connections 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/3357587741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08105eb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 -- 172.21.15.121:0/3357587741 >> 172.21.15.121:0/3357587741 conn(0x7f5e08101090 msgr2=0x7f5e081034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 -- 172.21.15.121:0/3357587741 shutdown_connections 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.626+0000 7f5e0ee62700 1 -- 172.21.15.121:0/3357587741 wait complete. 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0ee62700 1 Processor -- start 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0ee62700 1 -- start start 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0ee62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0ee62700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e08199c50 con 0x7f5e08105a90 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0cbfe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0cbfe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45802/0 (socket says 172.21.15.121:45802) 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0cbfe700 1 -- 172.21.15.121:0/1175020421 learned_addr learned my addr 172.21.15.121:0/1175020421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0cbfe700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc009550 con 0x7f5e08105a90 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.627+0000 7f5e0cbfe700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006af0 tx=0x7f5dfc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.628+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc0036a0 con 0x7f5e08105a90 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.628+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5dfc01a430 con 0x7f5e08105a90 2024-02-10T05:52:54.310 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.628+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5dfc0193c0 con 0x7f5e08105a90 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.628+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e08199e50 con 0x7f5e08105a90 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.628+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0819a2f0 con 0x7f5e08105a90 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f5dfc003800 con 0x7f5e08105a90 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e05ffb700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df803bb10 con 0x7f5df8038f40 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5dfc04cea0 con 0x7f5e08105a90 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e07fff700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.629+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.830+0000 7f5e07fff700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:48.830+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:49.231+0000 7f5e07fff700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:49.231+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-10T05:52:54.311 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:50.032+0000 7f5e07fff700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:50.032+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:51.632+0000 7f5e07fff700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2724527041 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:51.632+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:52.671+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 10) v1 ==== 45712+0+0 (secure 0 0 0) 0x7f5dfc019a40 con 0x7f5e08105a90 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:52.672+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 msgr2=0x7f5df803b400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:52.672+0000 7f5e05ffb700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.673+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 11) v1 ==== 45839+0+0 (secure 0 0 0) 0x7f5dfc04e850 con 0x7f5e08105a90 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.673+0000 7f5e05ffb700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 0x7f5df803ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.673+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5df803bb10 con 0x7f5df803c9f0 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.674+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 0x7f5df803ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.674+0000 7f5e07fff700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 0x7f5df803ede0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5df4003a10 tx=0x7f5df4009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.675+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f5df803bb10 con 0x7f5df803c9f0 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.681+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5e08063350 con 0x7f5df803c9f0 2024-02-10T05:52:54.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e05ffb700 1 -- 172.21.15.121:0/1175020421 <== mgr.14150 v2:172.21.15.121:6800/2329816948 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5e08063350 con 0x7f5df803c9f0 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 msgr2=0x7f5df803ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 0x7f5df803ede0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5df4003a10 tx=0x7f5df4009270 comp rx=0 tx=0).stop 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 msgr2=0x7f5e08199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc006af0 tx=0x7f5dfc011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 shutdown_connections 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5df803c9f0 0x7f5df803ede0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e08105a90 0x7f5e08199710 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 --2- 172.21.15.121:0/1175020421 >> [v2:172.21.15.121:6800/2724527041,v1:172.21.15.121:6801/2724527041] conn(0x7f5df8038f40 0x7f5df803b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 >> 172.21.15.121:0/1175020421 conn(0x7f5e08101090 msgr2=0x7f5e08101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 shutdown_connections 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:53.682+0000 7f5e0ee62700 1 -- 172.21.15.121:0/1175020421 wait complete. 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 9 is available 2024-02-10T05:52:54.313 INFO:teuthology.orchestra.run.smithi121.stdout:Generating a dashboard self-signed certificate... 2024-02-10T05:52:55.044 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:53] ENGINE Bus STARTING 2024-02-10T05:52:55.044 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:53] ENGINE Serving on http://172.21.15.121:8765 2024-02-10T05:52:55.044 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:52:55.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:52:55.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:53] ENGINE Serving on https://172.21.15.121:7150 2024-02-10T05:52:55.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:54 smithi121 ceph-mon[30020]: [10/Feb/2024:05:52:53] ENGINE Bus STARTED 2024-02-10T05:52:56.006 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:55 smithi121 ceph-mon[30020]: mgrmap e12: y(active, since 2s) 2024-02-10T05:52:56.006 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:52:56.006 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:52:56.006 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.145+0000 7fa7fe1a0700 1 Processor -- start 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.145+0000 7fa7fe1a0700 1 -- start start 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.145+0000 7fa7fe1a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.145+0000 7fa7fe1a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8105660 con 0x7fa7f8104c70 2024-02-10T05:52:56.016 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7fd19e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7fd19e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45864/0 (socket says 172.21.15.121:45864) 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7fd19e700 1 -- 172.21.15.121:0/358628712 learned_addr learned my addr 172.21.15.121:0/358628712 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7fd19e700 1 -- 172.21.15.121:0/358628712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8105990 con 0x7fa7f8104c70 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7fd19e700 1 --2- 172.21.15.121:0/358628712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40098a0 tx=0x7fa7f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c194a8b107653e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7effff700 1 -- 172.21.15.121:0/358628712 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4004160 con 0x7fa7f8104c70 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7effff700 1 -- 172.21.15.121:0/358628712 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa7f40042c0 con 0x7fa7f8104c70 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.146+0000 7fa7effff700 1 -- 172.21.15.121:0/358628712 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f4011850 con 0x7fa7f8104c70 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/358628712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 msgr2=0x7fa7f8105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/358628712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40098a0 tx=0x7fa7f4009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/358628712 shutdown_connections 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/358628712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f8105090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/358628712 >> 172.21.15.121:0/358628712 conn(0x7fa7f80ff530 msgr2=0x7fa7f8101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:56.017 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/358628712 shutdown_connections 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.147+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/358628712 wait complete. 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fe1a0700 1 Processor -- start 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fe1a0700 1 -- start start 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fe1a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fe1a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f818fcc0 con 0x7fa7f8104c70 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fd19e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fd19e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45868/0 (socket says 172.21.15.121:45868) 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fd19e700 1 -- 172.21.15.121:0/3556835494 learned_addr learned my addr 172.21.15.121:0/3556835494 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fd19e700 1 -- 172.21.15.121:0/3556835494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f4009550 con 0x7fa7f8104c70 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.148+0000 7fa7fd19e700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006af0 tx=0x7fa7f4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.149+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f40036a0 con 0x7fa7f8104c70 2024-02-10T05:52:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.149+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa7f401a430 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.149+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f818deb0 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.149+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f818e350 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.149+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7f40193c0 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f818ed60 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7fa7f4019630 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7ee7fc700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 0x7fa7e403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa7f404d130 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7fc99d700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 0x7fa7e403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.150+0000 7fa7fc99d700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 0x7fa7e403b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8006fd0 tx=0x7fa7e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.156+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa7f4018350 con 0x7fa7f8104c70 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.365+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7f818e590 con 0x7fa7e4038ee0 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.464+0000 7fa7ee7fc700 1 -- 172.21.15.121:0/3556835494 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa7f818e590 con 0x7fa7e4038ee0 2024-02-10T05:52:56.019 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 msgr2=0x7fa7e403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 0x7fa7e403b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8006fd0 tx=0x7fa7e8006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 msgr2=0x7fa7f818f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4006af0 tx=0x7fa7f4011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 shutdown_connections 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa7e4038ee0 0x7fa7e403b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.466+0000 7fa7fe1a0700 1 --2- 172.21.15.121:0/3556835494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7f8104c70 0x7fa7f818f780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.467+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 >> 172.21.15.121:0/3556835494 conn(0x7fa7f80ff530 msgr2=0x7fa7f8100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.467+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 shutdown_connections 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:55.467+0000 7fa7fe1a0700 1 -- 172.21.15.121:0/3556835494 wait complete. 2024-02-10T05:52:56.020 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial admin user... 2024-02-10T05:52:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:56 smithi121 ceph-mon[30020]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:52:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:57 smithi121 ceph-mon[30020]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:52:58.154 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$cGqDhTYi2Vcba2aR02N9X.3cpVwRxG.I7P.gRs4H0ErS/oPW2Fpri", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707544376, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T05:52:58.154 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e6d479700 1 Processor -- start 2024-02-10T05:52:58.154 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e6d479700 1 -- start start 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e6d479700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e6d479700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e68104740 con 0x7f4e68103d50 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e66ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e66ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45874/0 (socket says 172.21.15.121:45874) 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e66ffd700 1 -- 172.21.15.121:0/428931369 learned_addr learned my addr 172.21.15.121:0/428931369 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.509+0000 7f4e66ffd700 1 -- 172.21.15.121:0/428931369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e68104880 con 0x7f4e68103d50 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e66ffd700 1 --2- 172.21.15.121:0/428931369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e580098a0 tx=0x7f4e58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d161be7f8ad919d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e65ffb700 1 -- 172.21.15.121:0/428931369 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e58004160 con 0x7f4e68103d50 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e65ffb700 1 -- 172.21.15.121:0/428931369 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4e580042c0 con 0x7f4e68103d50 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e65ffb700 1 -- 172.21.15.121:0/428931369 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e58011850 con 0x7f4e68103d50 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e6d479700 1 -- 172.21.15.121:0/428931369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 msgr2=0x7f4e68104170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:58.155 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.510+0000 7f4e6d479700 1 --2- 172.21.15.121:0/428931369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e580098a0 tx=0x7f4e58009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 -- 172.21.15.121:0/428931369 shutdown_connections 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 --2- 172.21.15.121:0/428931369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68103d50 0x7f4e68104170 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 -- 172.21.15.121:0/428931369 >> 172.21.15.121:0/428931369 conn(0x7f4e680ff750 msgr2=0x7f4e68101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 -- 172.21.15.121:0/428931369 shutdown_connections 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 -- 172.21.15.121:0/428931369 wait complete. 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 Processor -- start 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.511+0000 7f4e6d479700 1 -- start start 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e6d479700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e6d479700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6819a2d0 con 0x7f4e68199970 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e66ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e66ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45886/0 (socket says 172.21.15.121:45886) 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e66ffd700 1 -- 172.21.15.121:0/3734838621 learned_addr learned my addr 172.21.15.121:0/3734838621 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:58.156 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e66ffd700 1 -- 172.21.15.121:0/3734838621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58009550 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e66ffd700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e58003ec0 tx=0x7f4e58011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e580036a0 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4e5801f070 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6819a4d0 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e58026cf0 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.512+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6819d040 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.513+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f4e5801a440 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.514+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e68109fe0 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.514+0000 7f4e4ffff700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 0x7f4e5003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.514+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e58021070 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.514+0000 7f4e667fc700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 0x7f4e5003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.514+0000 7f4e667fc700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 0x7f4e5003b400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c006fd0 tx=0x7f4e5c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.519+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4e58025990 con 0x7f4e68199970 2024-02-10T05:52:58.157 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.753+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4e68063170 con 0x7f4e50038f40 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.981+0000 7f4e4ffff700 1 -- 172.21.15.121:0/3734838621 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f4e68063170 con 0x7f4e50038f40 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 msgr2=0x7f4e5003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 0x7f4e5003b400 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c006fd0 tx=0x7f4e5c006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 msgr2=0x7f4e68199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e58003ec0 tx=0x7f4e58011a50 comp rx=0 tx=0).stop 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 shutdown_connections 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f4e50038f40 0x7f4e5003b400 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 --2- 172.21.15.121:0/3734838621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e68199970 0x7f4e68199d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 >> 172.21.15.121:0/3734838621 conn(0x7f4e680ff750 msgr2=0x7f4e68100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 shutdown_connections 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:56.984+0000 7f4e6d479700 1 -- 172.21.15.121:0/3734838621 wait complete. 2024-02-10T05:52:58.158 INFO:teuthology.orchestra.run.smithi121.stdout:Fetching dashboard port number... 2024-02-10T05:52:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/20833351' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T05:52:59.377 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe67b1fb700 1 Processor -- start 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe67b1fb700 1 -- start start 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe67b1fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe67b1fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741069f0 con 0x7fe674106000 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe678f97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe678f97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34516/0 (socket says 172.21.15.121:34516) 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.616+0000 7fe678f97700 1 -- 172.21.15.121:0/1937441312 learned_addr learned my addr 172.21.15.121:0/1937441312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe678f97700 1 -- 172.21.15.121:0/1937441312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674106b30 con 0x7fe674106000 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe678f97700 1 --2- 172.21.15.121:0/1937441312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6700098a0 tx=0x7fe670009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2162c436b0f540cc server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:59.378 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe66b7fe700 1 -- 172.21.15.121:0/1937441312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe670004160 con 0x7fe674106000 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe66b7fe700 1 -- 172.21.15.121:0/1937441312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe6700042c0 con 0x7fe674106000 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe66b7fe700 1 -- 172.21.15.121:0/1937441312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe670011850 con 0x7fe674106000 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe67b1fb700 1 -- 172.21.15.121:0/1937441312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 msgr2=0x7fe674106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.617+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/1937441312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe6700098a0 tx=0x7fe670009bb0 comp rx=0 tx=0).stop 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- 172.21.15.121:0/1937441312 shutdown_connections 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/1937441312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe674106420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- 172.21.15.121:0/1937441312 >> 172.21.15.121:0/1937441312 conn(0x7fe6740ff740 msgr2=0x7fe674101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- 172.21.15.121:0/1937441312 shutdown_connections 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- 172.21.15.121:0/1937441312 wait complete. 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 Processor -- start 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- start start 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe67b1fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67419be60 con 0x7fe674106000 2024-02-10T05:52:59.379 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe678f97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe678f97700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34528/0 (socket says 172.21.15.121:34528) 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.618+0000 7fe678f97700 1 -- 172.21.15.121:0/20833351 learned_addr learned my addr 172.21.15.121:0/20833351 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe678f97700 1 -- 172.21.15.121:0/20833351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009550 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe678f97700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe670003ec0 tx=0x7fe670011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6700037d0 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe670003930 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67419c060 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6700193c0 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.619+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67419c500 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.620+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7fe670019540 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.620+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6741046f0 con 0x7fe674106000 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.620+0000 7fe669ffb700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 0x7fe65c03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:52:59.380 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.620+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe67004d0c0 con 0x7fe674106000 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.620+0000 7fe66bfff700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 0x7fe65c03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.621+0000 7fe66bfff700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 0x7fe65c03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe664006fd0 tx=0x7fe664006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.625+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe670018940 con 0x7fe674106000 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.821+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fe674195090 con 0x7fe674106000 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.821+0000 7fe669ffb700 1 -- 172.21.15.121:0/20833351 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fe670018350 con 0x7fe674106000 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 msgr2=0x7fe65c03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 0x7fe65c03b450 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe664006fd0 tx=0x7fe664006e40 comp rx=0 tx=0).stop 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 msgr2=0x7fe67419b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe670003ec0 tx=0x7fe670011980 comp rx=0 tx=0).stop 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 shutdown_connections 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe65c038f90 0x7fe65c03b450 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 --2- 172.21.15.121:0/20833351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe674106000 0x7fe67419b920 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 >> 172.21.15.121:0/20833351 conn(0x7fe6740ff740 msgr2=0x7fe674192150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:52:59.381 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 shutdown_connections 2024-02-10T05:52:59.382 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:58.823+0000 7fe67b1fb700 1 -- 172.21.15.121:0/20833351 wait complete. 2024-02-10T05:52:59.384 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:52:59.384 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-02-10T05:52:59.392 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:52:59.392 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-02-10T05:52:59.392 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-02-10T05:52:59.392 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T05:52:59.393 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph Dashboard is now available at: 2024-02-10T05:52:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:52:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: URL: https://smithi121.front.sepia.ceph.com:8443/ 2024-02-10T05:52:59.393 INFO:teuthology.orchestra.run.smithi121.stdout: User: admin 2024-02-10T05:52:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: Password: dgj32pb78y 2024-02-10T05:52:59.394 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:52:59.394 INFO:teuthology.orchestra.run.smithi121.stdout:Saving cluster configuration to /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config directory 2024-02-10T05:52:59.394 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling autotune for osd_memory_target 2024-02-10T05:53:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:52:59 smithi121 ceph-mon[30020]: mgrmap e13: y(active, since 6s) 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f01682e8700 1 Processor -- start 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f01682e8700 1 -- start start 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f01682e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f01682e8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160105380 con 0x7f0160104990 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f0166084700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f0166084700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34530/0 (socket says 172.21.15.121:34530) 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.834+0000 7f0166084700 1 -- 172.21.15.121:0/1601077373 learned_addr learned my addr 172.21.15.121:0/1601077373 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f0166084700 1 -- 172.21.15.121:0/1601077373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0160105b90 con 0x7f0160104990 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f0166084700 1 --2- 172.21.15.121:0/1601077373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f015c00bb40 tx=0x7f015c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82347e617fdabdc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:00.642 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f0165082700 1 -- 172.21.15.121:0/1601077373 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f015c00dbe0 con 0x7f0160104990 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f0165082700 1 -- 172.21.15.121:0/1601077373 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f015c0044b0 con 0x7f0160104990 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f0165082700 1 -- 172.21.15.121:0/1601077373 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f015c003cd0 con 0x7f0160104990 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f01682e8700 1 -- 172.21.15.121:0/1601077373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 msgr2=0x7f0160104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.835+0000 7f01682e8700 1 --2- 172.21.15.121:0/1601077373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f015c00bb40 tx=0x7f015c00be50 comp rx=0 tx=0).stop 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- 172.21.15.121:0/1601077373 shutdown_connections 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 --2- 172.21.15.121:0/1601077373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f0160104db0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- 172.21.15.121:0/1601077373 >> 172.21.15.121:0/1601077373 conn(0x7f01600fffd0 msgr2=0x7f01601023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- 172.21.15.121:0/1601077373 shutdown_connections 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- 172.21.15.121:0/1601077373 wait complete. 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 Processor -- start 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- start start 2024-02-10T05:53:00.643 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f01682e8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0160197a20 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f0166084700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f0166084700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34534/0 (socket says 172.21.15.121:34534) 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.836+0000 7f0166084700 1 -- 172.21.15.121:0/1136527957 learned_addr learned my addr 172.21.15.121:0/1136527957 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f0166084700 1 -- 172.21.15.121:0/1136527957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c00b7f0 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f0166084700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f015c003a40 tx=0x7f015c00dfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f015c01e070 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f015c010b00 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0160197c20 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f015c022970 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.837+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01601980c0 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.838+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0160063350 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.838+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f015c010c70 con 0x7f0160104990 2024-02-10T05:53:00.644 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.838+0000 7f01577fe700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 0x7f014c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.838+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f015c04f6a0 con 0x7f0160104990 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.838+0000 7f0165883700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 0x7f014c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.839+0000 7f0165883700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 0x7f014c03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0150006fd0 tx=0x7f0150006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:52:59.843+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f015c021c40 con 0x7f0160104990 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.035+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0160190c50 con 0x7f0160104990 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.036+0000 7f01577fe700 1 -- 172.21.15.121:0/1136527957 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f015c027030 con 0x7f0160104990 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.037+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 msgr2=0x7f014c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.037+0000 7f01682e8700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 0x7f014c03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0150006fd0 tx=0x7f0150006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 msgr2=0x7f01601974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f015c003a40 tx=0x7f015c00dfd0 comp rx=0 tx=0).stop 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 shutdown_connections 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f014c039020 0x7f014c03b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 --2- 172.21.15.121:0/1136527957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0160104990 0x7f01601974e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:00.645 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 >> 172.21.15.121:0/1136527957 conn(0x7f01600fffd0 msgr2=0x7f016018dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:00.646 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 shutdown_connections 2024-02-10T05:53:00.646 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:00.038+0000 7f01682e8700 1 -- 172.21.15.121:0/1136527957 wait complete. 2024-02-10T05:53:02.040 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f89b8700 1 Processor -- start 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f89b8700 1 -- start start 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f89b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f89b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40796e0 con 0x7fb4f4106560 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34540/0 (socket says 172.21.15.121:34540) 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.121+0000 7fb4f259c700 1 -- 172.21.15.121:0/879759701 learned_addr learned my addr 172.21.15.121:0/879759701 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.122+0000 7fb4f259c700 1 -- 172.21.15.121:0/879759701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4108e90 con 0x7fb4f4106560 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.122+0000 7fb4f259c700 1 --2- 172.21.15.121:0/879759701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40098a0 tx=0x7fb4e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14f287c829cd0ad server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.122+0000 7fb4f1d9b700 1 -- 172.21.15.121:0/879759701 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e4004160 con 0x7fb4f4106560 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.122+0000 7fb4f1d9b700 1 -- 172.21.15.121:0/879759701 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb4e40042c0 con 0x7fb4f4106560 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.122+0000 7fb4f1d9b700 1 -- 172.21.15.121:0/879759701 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e4011850 con 0x7fb4f4106560 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 -- 172.21.15.121:0/879759701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 msgr2=0x7fb4f4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/879759701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4e40098a0 tx=0x7fb4e4009bb0 comp rx=0 tx=0).stop 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 -- 172.21.15.121:0/879759701 shutdown_connections 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/879759701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4108950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 -- 172.21.15.121:0/879759701 >> 172.21.15.121:0/879759701 conn(0x7fb4f40fff70 msgr2=0x7fb4f4102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 -- 172.21.15.121:0/879759701 shutdown_connections 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 -- 172.21.15.121:0/879759701 wait complete. 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.123+0000 7fb4f89b8700 1 Processor -- start 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f89b8700 1 -- start start 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f89b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:02.066 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f89b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f4199c60 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34552/0 (socket says 172.21.15.121:34552) 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f259c700 1 -- 172.21.15.121:0/1754709859 learned_addr learned my addr 172.21.15.121:0/1754709859 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f259c700 1 -- 172.21.15.121:0/1754709859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4009550 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f259c700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4003ec0 tx=0x7fb4e4011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e40037d0 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb4e4003930 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.124+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f4199e60 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.125+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4e40193c0 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.125+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f419a300 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fb4e4019630 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f4109bb0 con 0x7fb4f4106560 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4ebfff700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 0x7fb4e003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:02.067 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb4e404d420 con 0x7fb4f4106560 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4e9d9b700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 0x7fb4e003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.126+0000 7fb4e9d9b700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 0x7fb4e003b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc006fd0 tx=0x7fb4dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.131+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4e400b3e0 con 0x7fb4f4106560 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.433+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb4f419cce0 con 0x7fb4f4106560 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.435+0000 7fb4ebfff700 1 -- 172.21.15.121:0/1754709859 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb4e4018350 con 0x7fb4f4106560 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 msgr2=0x7fb4e003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 0x7fb4e003b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc006fd0 tx=0x7fb4dc006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 msgr2=0x7fb4f4199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4003ec0 tx=0x7fb4e4011980 comp rx=0 tx=0).stop 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 shutdown_connections 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb4e0038fd0 0x7fb4e003b490 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 --2- 172.21.15.121:0/1754709859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f4106560 0x7fb4f4199720 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:02.068 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 >> 172.21.15.121:0/1754709859 conn(0x7fb4f40fff70 msgr2=0x7fb4f41009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 shutdown_connections 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-02-10T05:53:01.437+0000 7fb4f89b8700 1 -- 172.21.15.121:0/1754709859 wait complete. 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: ceph telemetry on 2024-02-10T05:53:02.069 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.070 INFO:teuthology.orchestra.run.smithi121.stdout:For more information see: 2024-02-10T05:53:02.070 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.070 INFO:teuthology.orchestra.run.smithi121.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-10T05:53:02.070 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:02.070 INFO:teuthology.orchestra.run.smithi121.stdout:Bootstrap complete. 2024-02-10T05:53:02.078 INFO:tasks.cephadm:Fetching config... 2024-02-10T05:53:02.078 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:02.078 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T05:53:02.095 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T05:53:02.095 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:02.095 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T05:53:02.154 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T05:53:02.155 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:02.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T05:53:02.222 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T05:53:02.222 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:02.223 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T05:53:02.276 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T05:53:02.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaKIT4FgRpaqI5IS5n63XmrLL5oVkAPmRzZbspKpqvVZOFgiyTXmWQrEWQ/ZyRt3lWHvoOIoFEY1ME8eXlxEUzCGf3vzYDCGKUHSyDBKzlQmN151Iz0YVuFhvrycp1voQCBbH1vvxm1gSZjq3bT+p/AC2zzV32p4ybTDKVOmCNN0aKMeVVl+V+D+9egbzaEaz852d1FPIvW64cPVGApbdyd5Qn2bN/NNkQgkJBuGfI2UnF71Siuarm/ulT9vdhjraikBhVhC1OkqHukerprXbzinPkiXZ67Pbhj+9ILTwikYgzgErsb8Gns5Yh0Aw5SMDMfKKPRzWVVMgsQv4xOHE9/3cdta/yrgres7X+NYeEz3iEpTlp5WHnwXAHHm2qaa9gjhF94kFlSe6YV143i2GPYrSWC9vvImpQuTZ9cw+QltlOvysO7pbbkVBHe4bwSDC9JrTFomxNdBC/UEkgaO9r8I24Gxy39OVnicVHjir3P6jFWNnqqSTBJDDDcrslYT0= ceph-616c67f8-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:53:02.355 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaKIT4FgRpaqI5IS5n63XmrLL5oVkAPmRzZbspKpqvVZOFgiyTXmWQrEWQ/ZyRt3lWHvoOIoFEY1ME8eXlxEUzCGf3vzYDCGKUHSyDBKzlQmN151Iz0YVuFhvrycp1voQCBbH1vvxm1gSZjq3bT+p/AC2zzV32p4ybTDKVOmCNN0aKMeVVl+V+D+9egbzaEaz852d1FPIvW64cPVGApbdyd5Qn2bN/NNkQgkJBuGfI2UnF71Siuarm/ulT9vdhjraikBhVhC1OkqHukerprXbzinPkiXZ67Pbhj+9ILTwikYgzgErsb8Gns5Yh0Aw5SMDMfKKPRzWVVMgsQv4xOHE9/3cdta/yrgres7X+NYeEz3iEpTlp5WHnwXAHHm2qaa9gjhF94kFlSe6YV143i2GPYrSWC9vvImpQuTZ9cw+QltlOvysO7pbbkVBHe4bwSDC9JrTFomxNdBC/UEkgaO9r8I24Gxy39OVnicVHjir3P6jFWNnqqSTBJDDDcrslYT0= ceph-616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:02.368 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaKIT4FgRpaqI5IS5n63XmrLL5oVkAPmRzZbspKpqvVZOFgiyTXmWQrEWQ/ZyRt3lWHvoOIoFEY1ME8eXlxEUzCGf3vzYDCGKUHSyDBKzlQmN151Iz0YVuFhvrycp1voQCBbH1vvxm1gSZjq3bT+p/AC2zzV32p4ybTDKVOmCNN0aKMeVVl+V+D+9egbzaEaz852d1FPIvW64cPVGApbdyd5Qn2bN/NNkQgkJBuGfI2UnF71Siuarm/ulT9vdhjraikBhVhC1OkqHukerprXbzinPkiXZ67Pbhj+9ILTwikYgzgErsb8Gns5Yh0Aw5SMDMfKKPRzWVVMgsQv4xOHE9/3cdta/yrgres7X+NYeEz3iEpTlp5WHnwXAHHm2qaa9gjhF94kFlSe6YV143i2GPYrSWC9vvImpQuTZ9cw+QltlOvysO7pbbkVBHe4bwSDC9JrTFomxNdBC/UEkgaO9r8I24Gxy39OVnicVHjir3P6jFWNnqqSTBJDDDcrslYT0= ceph-616c67f8-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:53:02.413 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaKIT4FgRpaqI5IS5n63XmrLL5oVkAPmRzZbspKpqvVZOFgiyTXmWQrEWQ/ZyRt3lWHvoOIoFEY1ME8eXlxEUzCGf3vzYDCGKUHSyDBKzlQmN151Iz0YVuFhvrycp1voQCBbH1vvxm1gSZjq3bT+p/AC2zzV32p4ybTDKVOmCNN0aKMeVVl+V+D+9egbzaEaz852d1FPIvW64cPVGApbdyd5Qn2bN/NNkQgkJBuGfI2UnF71Siuarm/ulT9vdhjraikBhVhC1OkqHukerprXbzinPkiXZ67Pbhj+9ILTwikYgzgErsb8Gns5Yh0Aw5SMDMfKKPRzWVVMgsQv4xOHE9/3cdta/yrgres7X+NYeEz3iEpTlp5WHnwXAHHm2qaa9gjhF94kFlSe6YV143i2GPYrSWC9vvImpQuTZ9cw+QltlOvysO7pbbkVBHe4bwSDC9JrTFomxNdBC/UEkgaO9r8I24Gxy39OVnicVHjir3P6jFWNnqqSTBJDDDcrslYT0= ceph-616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:02.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T05:53:02.627 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1754709859' entity='client.admin' 2024-02-10T05:53:02.699 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/1239909811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 msgr2=0x7f9d54100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/1239909811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d54100a60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009910 tx=0x7f9d44009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/1239909811 shutdown_connections 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/1239909811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d54100a60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/1239909811 >> 172.21.15.121:0/1239909811 conn(0x7f9d540fbc20 msgr2=0x7f9d540fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:05.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/1239909811 shutdown_connections 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/1239909811 wait complete. 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 Processor -- start 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.442+0000 7f9d5a4c8700 1 -- start start 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d5a4c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d5a4c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5406a970 con 0x7f9d54100640 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34570/0 (socket says 172.21.15.121:34570) 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d53fff700 1 -- 172.21.15.121:0/2392184334 learned_addr learned my addr 172.21.15.121:0/2392184334 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d53fff700 1 -- 172.21.15.121:0/2392184334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440095f0 con 0x7f9d54100640 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d53fff700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d44005660 tx=0x7f9d44005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d4400b960 con 0x7f9d54100640 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d54198f30 con 0x7f9d54100640 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9d4400f460 con 0x7f9d54100640 2024-02-10T05:53:05.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.443+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d541993d0 con 0x7f9d54100640 2024-02-10T05:53:05.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.444+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d44017440 con 0x7f9d54100640 2024-02-10T05:53:05.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.444+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f9d440176b0 con 0x7f9d54100640 2024-02-10T05:53:05.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.444+0000 7f9d517fa700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 0x7f9d3c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:05.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.445+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d4404dfb0 con 0x7f9d54100640 2024-02-10T05:53:05.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.445+0000 7f9d537fe700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 0x7f9d3c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:05.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.445+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5418eec0 con 0x7f9d54100640 2024-02-10T05:53:05.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.445+0000 7f9d537fe700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 0x7f9d3c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d48006fd0 tx=0x7f9d48006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.450+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d44027020 con 0x7f9d54100640 2024-02-10T05:53:05.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.654+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9d54106240 con 0x7f9d54100640 2024-02-10T05:53:05.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.656+0000 7f9d517fa700 1 -- 172.21.15.121:0/2392184334 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9d4402c440 con 0x7f9d54100640 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 msgr2=0x7f9d3c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 0x7f9d3c03b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d48006fd0 tx=0x7f9d48006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 msgr2=0x7f9d5406a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d44005660 tx=0x7f9d44005740 comp rx=0 tx=0).stop 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 shutdown_connections 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f9d3c038fe0 0x7f9d3c03b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 --2- 172.21.15.121:0/2392184334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d54100640 0x7f9d5406a430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 >> 172.21.15.121:0/2392184334 conn(0x7f9d540fbc20 msgr2=0x7f9d540fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 shutdown_connections 2024-02-10T05:53:05.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:05.659+0000 7f9d5a4c8700 1 -- 172.21.15.121:0/2392184334 wait complete. 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:05.672 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:06.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2392184334' entity='client.admin' 2024-02-10T05:53:06.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:06.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:06.787 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T05:53:06.787 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T05:53:07.023 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.615+0000 7f36472a5700 1 -- 172.21.15.121:0/2496402712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 msgr2=0x7f3640102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.615+0000 7f36472a5700 1 --2- 172.21.15.121:0/2496402712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f3640102090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f363c009910 tx=0x7f363c009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.615+0000 7f36472a5700 1 -- 172.21.15.121:0/2496402712 shutdown_connections 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.615+0000 7f36472a5700 1 --2- 172.21.15.121:0/2496402712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f3640102090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 -- 172.21.15.121:0/2496402712 >> 172.21.15.121:0/2496402712 conn(0x7f36400fb340 msgr2=0x7f36400fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 -- 172.21.15.121:0/2496402712 shutdown_connections 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 -- 172.21.15.121:0/2496402712 wait complete. 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 Processor -- start 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 -- start start 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f36472a5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3640196720 con 0x7f3640101c70 2024-02-10T05:53:08.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.616+0000 7f3645041700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f3645041700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39926/0 (socket says 172.21.15.121:39926) 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f3645041700 1 -- 172.21.15.121:0/4237546856 learned_addr learned my addr 172.21.15.121:0/4237546856 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f3645041700 1 -- 172.21.15.121:0/4237546856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0095f0 con 0x7f3640101c70 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f3645041700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f363c005660 tx=0x7f363c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f363c00b960 con 0x7f3640101c70 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f363c00f460 con 0x7f3640101c70 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3640196920 con 0x7f3640101c70 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f363c017440 con 0x7f3640101c70 2024-02-10T05:53:08.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.617+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3640196dc0 con 0x7f3640101c70 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.618+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f363c017700 con 0x7f3640101c70 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.618+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364018ff00 con 0x7f3640101c70 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.618+0000 7f36367fc700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 0x7f362c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.618+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f363c04d690 con 0x7f3640101c70 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.618+0000 7f3644840700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 0x7f362c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:08.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.619+0000 7f3644840700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 0x7f362c03b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3630006fd0 tx=0x7f3630006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:08.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.623+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f363c015a40 con 0x7f3640101c70 2024-02-10T05:53:08.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.822+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f3640105ba0 con 0x7f362c038f80 2024-02-10T05:53:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.825+0000 7f36367fc700 1 -- 172.21.15.121:0/4237546856 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3640105ba0 con 0x7f362c038f80 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.827+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 msgr2=0x7f362c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.827+0000 7f36472a5700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 0x7f362c03b440 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3630006fd0 tx=0x7f3630006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.827+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 msgr2=0x7f36401961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.827+0000 7f36472a5700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f363c005660 tx=0x7f363c005740 comp rx=0 tx=0).stop 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 shutdown_connections 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f362c038f80 0x7f362c03b440 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 --2- 172.21.15.121:0/4237546856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3640101c70 0x7f36401961e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 >> 172.21.15.121:0/4237546856 conn(0x7f36400fb340 msgr2=0x7f3640103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 shutdown_connections 2024-02-10T05:53:08.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:08.828+0000 7f36472a5700 1 -- 172.21.15.121:0/4237546856 wait complete. 2024-02-10T05:53:09.569 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-02-10T05:53:09.569 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:53:09.569 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:53:09.587 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:53:09.588 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:09.644 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-02-10T05:53:09.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch host add smithi146 2024-02-10T05:53:09.880 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:10 smithi121 ceph-mon[30020]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:53:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:10 smithi121 ceph-mon[30020]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:10 smithi121 ceph-mon[30020]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:10 smithi121 ceph-mon[30020]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:53:11.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 -- 172.21.15.121:0/2799607334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 msgr2=0x7fc60c107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:11.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 --2- 172.21.15.121:0/2799607334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c107ac0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009910 tx=0x7fc5fc009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:11.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 -- 172.21.15.121:0/2799607334 shutdown_connections 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 --2- 172.21.15.121:0/2799607334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c107ac0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 -- 172.21.15.121:0/2799607334 >> 172.21.15.121:0/2799607334 conn(0x7fc60c0fd1f0 msgr2=0x7fc60c0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 -- 172.21.15.121:0/2799607334 shutdown_connections 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.440+0000 7fc612a07700 1 -- 172.21.15.121:0/2799607334 wait complete. 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.441+0000 7fc612a07700 1 Processor -- start 2024-02-10T05:53:11.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.441+0000 7fc612a07700 1 -- start start 2024-02-10T05:53:11.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.441+0000 7fc612a07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.441+0000 7fc612a07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60c196ac0 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc60bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc60bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39938/0 (socket says 172.21.15.121:39938) 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc60bfff700 1 -- 172.21.15.121:0/247899845 learned_addr learned my addr 172.21.15.121:0/247899845 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc60bfff700 1 -- 172.21.15.121:0/247899845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0095f0 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc60bfff700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006010 tx=0x7fc5fc005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5fc01d070 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60c196d20 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc5fc00b6f0 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5fc003810 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.442+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60c1971a0 con 0x7fc60c1076e0 2024-02-10T05:53:11.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.443+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fc5fc003a50 con 0x7fc60c1076e0 2024-02-10T05:53:11.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.443+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc60c063350 con 0x7fc60c1076e0 2024-02-10T05:53:11.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.443+0000 7fc6097fa700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 0x7fc5f403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:11.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.443+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc5fc04d050 con 0x7fc60c1076e0 2024-02-10T05:53:11.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.443+0000 7fc60b7fe700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 0x7fc5f403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:11.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.444+0000 7fc60b7fe700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 0x7fc5f403b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:11.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.448+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc5fc029360 con 0x7fc60c1076e0 2024-02-10T05:53:11.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:11.645+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}) v1 -- 0x7fc60c10b020 con 0x7fc5f4038f80 2024-02-10T05:53:12.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:11 smithi121 ceph-mon[30020]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:12 smithi121 ceph-mon[30020]: Deploying cephadm binary to smithi146 2024-02-10T05:53:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:12 smithi121 ceph-mon[30020]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:15.937 INFO:teuthology.orchestra.run.smithi121.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-02-10T05:53:15.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.936+0000 7fc6097fa700 1 -- 172.21.15.121:0/247899845 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc60c10b020 con 0x7fc5f4038f80 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 msgr2=0x7fc5f403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 0x7fc5f403b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 msgr2=0x7fc60c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc006010 tx=0x7fc5fc005710 comp rx=0 tx=0).stop 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 shutdown_connections 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fc5f4038f80 0x7fc5f403b440 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 --2- 172.21.15.121:0/247899845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc60c1076e0 0x7fc60c196580 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 >> 172.21.15.121:0/247899845 conn(0x7fc60c0fd1f0 msgr2=0x7fc60c109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 shutdown_connections 2024-02-10T05:53:15.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:15.938+0000 7fc612a07700 1 -- 172.21.15.121:0/247899845 wait complete. 2024-02-10T05:53:16.031 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:15 smithi121 ceph-mon[30020]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:16.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:53:16.789 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:17.197 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:16 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:17.197 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:16 smithi121 ceph-mon[30020]: Added host smithi146 2024-02-10T05:53:17.197 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:16 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:17.197 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:16 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:17.197 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:16 smithi121 ceph-mon[30020]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:18.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- 172.21.15.121:0/3181859092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 msgr2=0x7fe490102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 --2- 172.21.15.121:0/3181859092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490102ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009910 tx=0x7fe48c009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- 172.21.15.121:0/3181859092 shutdown_connections 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 --2- 172.21.15.121:0/3181859092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490102ca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- 172.21.15.121:0/3181859092 >> 172.21.15.121:0/3181859092 conn(0x7fe4900fc130 msgr2=0x7fe4900fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- 172.21.15.121:0/3181859092 shutdown_connections 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- 172.21.15.121:0/3181859092 wait complete. 2024-02-10T05:53:18.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 Processor -- start 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- start start 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.474+0000 7fe498338700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490196b60 con 0x7fe4901028c0 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4960d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4960d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55498/0 (socket says 172.21.15.121:55498) 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4960d4700 1 -- 172.21.15.121:0/167927486 learned_addr learned my addr 172.21.15.121:0/167927486 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:18.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4960d4700 1 -- 172.21.15.121:0/167927486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c0095f0 con 0x7fe4901028c0 2024-02-10T05:53:18.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4960d4700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0055f0 tx=0x7fe48c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:18.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe48c01d070 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe48c00b750 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe490196d60 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe48c0037b0 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.475+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe490197180 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.476+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fe48c003a20 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.476+0000 7fe4877fe700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 0x7fe47c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.476+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490063350 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.476+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe48c04cd50 con 0x7fe4901028c0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.476+0000 7fe4958d3700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 0x7fe47c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:18.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.477+0000 7fe4958d3700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 0x7fe47c03b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe480006fd0 tx=0x7fe480006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:18.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.481+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe48c026080 con 0x7fe4901028c0 2024-02-10T05:53:18.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.678+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe49010b0a0 con 0x7fe47c038fe0 2024-02-10T05:53:18.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.679+0000 7fe4877fe700 1 -- 172.21.15.121:0/167927486 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fe49010b0a0 con 0x7fe47c038fe0 2024-02-10T05:53:18.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:53:18.680 INFO:teuthology.orchestra.run.smithi121.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 msgr2=0x7fe47c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 0x7fe47c03b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe480006fd0 tx=0x7fe480006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 msgr2=0x7fe490196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0055f0 tx=0x7fe48c0056d0 comp rx=0 tx=0).stop 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 shutdown_connections 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe47c038fe0 0x7fe47c03b4a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 --2- 172.21.15.121:0/167927486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4901028c0 0x7fe490196620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 >> 172.21.15.121:0/167927486 conn(0x7fe4900fc130 msgr2=0x7fe4900fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 shutdown_connections 2024-02-10T05:53:18.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:18.681+0000 7fe498338700 1 -- 172.21.15.121:0/167927486 wait complete. 2024-02-10T05:53:19.301 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T05:53:19.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T05:53:19.537 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:19.928 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:19 smithi121 ceph-mon[30020]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:19.928 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:19 smithi121 ceph-mon[30020]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:53:19.928 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 -- 172.21.15.121:0/3014248328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001066a0 msgr2=0x7f1700106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 --2- 172.21.15.121:0/3014248328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001066a0 0x7f1700106a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f16f800b0a0 tx=0x7f16f800b3b0 comp rx=0 tx=0).stop 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 -- 172.21.15.121:0/3014248328 shutdown_connections 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 --2- 172.21.15.121:0/3014248328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17001066a0 0x7f1700106a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 -- 172.21.15.121:0/3014248328 >> 172.21.15.121:0/3014248328 conn(0x7f17000fc120 msgr2=0x7f17000fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.210+0000 7f1707b83700 1 -- 172.21.15.121:0/3014248328 shutdown_connections 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f1707b83700 1 -- 172.21.15.121:0/3014248328 wait complete. 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f1707b83700 1 Processor -- start 2024-02-10T05:53:21.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f1707b83700 1 -- start start 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f1707b83700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f1707b83700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17000fe470 con 0x7f17000fc5d0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f170591f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f170591f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55508/0 (socket says 172.21.15.121:55508) 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.211+0000 7f170591f700 1 -- 172.21.15.121:0/3742251514 learned_addr learned my addr 172.21.15.121:0/3742251514 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f170591f700 1 -- 172.21.15.121:0/3742251514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8009d00 con 0x7f17000fc5d0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f170591f700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16f8015040 tx=0x7f16f800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f8021070 con 0x7f17000fc5d0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f16f8003c10 con 0x7f17000fc5d0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f16f800e040 con 0x7f17000fc5d0 2024-02-10T05:53:21.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17000fcef0 con 0x7f17000fc5d0 2024-02-10T05:53:21.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.212+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17000fd370 con 0x7f17000fc5d0 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f16f8019070 con 0x7f17000fc5d0 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1700063350 con 0x7f17000fc5d0 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f16feffd700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 0x7f16e803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16f804c8f0 con 0x7f17000fc5d0 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f170511e700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 0x7f16e803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:21.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.213+0000 7f170511e700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 0x7f16e803b4a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f16f4006fd0 tx=0x7f16f4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:21.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.218+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f16f801ceb0 con 0x7f17000fc5d0 2024-02-10T05:53:21.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.415+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f17000fdab0 con 0x7f17000fc5d0 2024-02-10T05:53:21.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.826+0000 7f16feffd700 1 -- 172.21.15.121:0/3742251514 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f16f8017020 con 0x7f17000fc5d0 2024-02-10T05:53:21.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 msgr2=0x7f16e803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:21.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 0x7f16e803b4a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f16f4006fd0 tx=0x7f16f4006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 msgr2=0x7f17000fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16f8015040 tx=0x7f16f800bd40 comp rx=0 tx=0).stop 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 shutdown_connections 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f16e8038fe0 0x7f16e803b4a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 --2- 172.21.15.121:0/3742251514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17000fc5d0 0x7f17000fc9b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 >> 172.21.15.121:0/3742251514 conn(0x7f17000fc120 msgr2=0x7f170006a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 shutdown_connections 2024-02-10T05:53:21.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:21.828+0000 7f1707b83700 1 -- 172.21.15.121:0/3742251514 wait complete. 2024-02-10T05:53:21.830 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-02-10T05:53:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:21 smithi121 ceph-mon[30020]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:53:22.479 INFO:tasks.cephadm:Adding mon.a on smithi121 2024-02-10T05:53:22.480 INFO:tasks.cephadm:Adding mon.c on smithi121 2024-02-10T05:53:22.480 INFO:tasks.cephadm:Adding mon.b on smithi146 2024-02-10T05:53:22.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b' 2024-02-10T05:53:22.719 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:22 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 -- 172.21.15.146:0/1605215640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 msgr2=0x7f3938106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 --2- 172.21.15.146:0/1605215640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938106a70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3928009910 tx=0x7f3928009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 -- 172.21.15.146:0/1605215640 shutdown_connections 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 --2- 172.21.15.146:0/1605215640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938106a70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 -- 172.21.15.146:0/1605215640 >> 172.21.15.146:0/1605215640 conn(0x7f39380fc130 msgr2=0x7f39380fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 -- 172.21.15.146:0/1605215640 shutdown_connections 2024-02-10T05:53:24.246 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.245+0000 7f393e6f9700 1 -- 172.21.15.146:0/1605215640 wait complete. 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.246+0000 7f393e6f9700 1 Processor -- start 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.246+0000 7f393e6f9700 1 -- start start 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.246+0000 7f393e6f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.246+0000 7f393e6f9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39381959d0 con 0x7f3938106690 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.246+0000 7f3937fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:24.248 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f3937fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.146:60076/0 (socket says 172.21.15.146:60076) 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f3937fff700 1 -- 172.21.15.146:0/2249774899 learned_addr learned my addr 172.21.15.146:0/2249774899 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f3937fff700 1 -- 172.21.15.146:0/2249774899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280095f0 con 0x7f3938106690 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f3937fff700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3928005660 tx=0x7f3928005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f392801d070 con 0x7f3938106690 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3938195c30 con 0x7f3938106690 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3938196130 con 0x7f3938106690 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f392800b6f0 con 0x7f3938106690 2024-02-10T05:53:24.249 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.247+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3928003680 con 0x7f3938106690 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.248+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938063350 con 0x7f3938106690 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.248+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f39280038f0 con 0x7f3938106690 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.248+0000 7f39357fa700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 0x7f392003d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.248+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f392804ccb0 con 0x7f3938106690 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.249+0000 7f39377fe700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 0x7f392003d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:24.250 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.249+0000 7f39377fe700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 0x7f392003d650 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f392c006fd0 tx=0x7f392c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:24.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.254+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3928029c30 con 0x7f3938106690 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:24.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.457+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f3938196130 con 0x7f392003b190 2024-02-10T05:53:24.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.463+0000 7f39357fa700 1 -- 172.21.15.146:0/2249774899 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3938196130 con 0x7f392003b190 2024-02-10T05:53:24.464 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-02-10T05:53:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.465+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 msgr2=0x7f392003d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.465+0000 7f393e6f9700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 0x7f392003d650 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f392c006fd0 tx=0x7f392c006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.465+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 msgr2=0x7f3938195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.465+0000 7f393e6f9700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3928005660 tx=0x7f3928005740 comp rx=0 tx=0).stop 2024-02-10T05:53:24.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 shutdown_connections 2024-02-10T05:53:24.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f392003b190 0x7f392003d650 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:24.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 --2- 172.21.15.146:0/2249774899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3938106690 0x7f3938195490 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:24.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 >> 172.21.15.146:0/2249774899 conn(0x7f39380fc130 msgr2=0x7f39380fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:24.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 shutdown_connections 2024-02-10T05:53:24.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:24.466+0000 7f393e6f9700 1 -- 172.21.15.146:0/2249774899 wait complete. 2024-02-10T05:53:25.103 DEBUG:teuthology.orchestra.run.smithi121:mon.c> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T05:53:25.106 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T05:53:25.108 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:53:25.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:53:25.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: Saving service mon spec with placement smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b;count:3 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: Deploying daemon mon.b on smithi146 2024-02-10T05:53:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:25 smithi121 ceph-mon[30020]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:25.851 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:27 smithi121 ceph-mon[30020]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:28.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 -- 172.21.15.146:0/3726738996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 msgr2=0x7fee78107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:28.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 --2- 172.21.15.146:0/3726738996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78107ac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee70009910 tx=0x7fee70009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:28.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 -- 172.21.15.146:0/3726738996 shutdown_connections 2024-02-10T05:53:28.715 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 --2- 172.21.15.146:0/3726738996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78107ac0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 -- 172.21.15.146:0/3726738996 >> 172.21.15.146:0/3726738996 conn(0x7fee780fd1f0 msgr2=0x7fee780ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 -- 172.21.15.146:0/3726738996 shutdown_connections 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.713+0000 7fee7dd18700 1 -- 172.21.15.146:0/3726738996 wait complete. 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee7dd18700 1 Processor -- start 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee7dd18700 1 -- start start 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee7dd18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee7dd18700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee78075f10 con 0x7fee781076e0 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee777fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee777fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.146:41770/0 (socket says 172.21.15.146:41770) 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.714+0000 7fee777fe700 1 -- 172.21.15.146:0/544003628 learned_addr learned my addr 172.21.15.146:0/544003628 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee777fe700 1 -- 172.21.15.146:0/544003628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700095f0 con 0x7fee781076e0 2024-02-10T05:53:28.716 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee777fe700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee70005660 tx=0x7fee70005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee7001d070 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee78074560 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fee7000b6f0 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee700037f0 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.715+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee780749e0 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.716+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fee70003a60 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.716+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee78105990 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.716+0000 7fee74ff9700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 0x7fee6003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.716+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fee7004cfb0 con 0x7fee781076e0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.716+0000 7fee76ffd700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 0x7fee6003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:28.718 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.717+0000 7fee76ffd700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 0x7fee6003b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee6c006fd0 tx=0x7fee6c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:28.722 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:28.721+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fee70029360 con 0x7fee781076e0 2024-02-10T05:53:29.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.050+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee78075550 con 0x7fee781076e0 2024-02-10T05:53:29.051 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.051+0000 7fee74ff9700 1 -- 172.21.15.146:0/544003628 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7fee70026030 con 0x7fee781076e0 2024-02-10T05:53:29.052 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:53:29.052 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:51:50.247436Z","created":"2024-02-10T05:51:50.247436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 msgr2=0x7fee6003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 0x7fee6003b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee6c006fd0 tx=0x7fee6c006e40 comp rx=0 tx=0).stop 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 msgr2=0x7fee78074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee70005660 tx=0x7fee70005740 comp rx=0 tx=0).stop 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 shutdown_connections 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fee60038f80 0x7fee6003b440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 --2- 172.21.15.146:0/544003628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee781076e0 0x7fee78074020 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 >> 172.21.15.146:0/544003628 conn(0x7fee780fd1f0 msgr2=0x7fee781095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 shutdown_connections 2024-02-10T05:53:29.054 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:29.053+0000 7fee7dd18700 1 -- 172.21.15.146:0/544003628 wait complete. 2024-02-10T05:53:29.055 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-02-10T05:53:29.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:28 smithi121 ceph-mon[30020]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:29.362 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:29 smithi146 systemd[1]: Starting Ceph mon.b for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:53:29.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:29 smithi146 podman[33757]: 2024-02-10 05:53:29.784032408 +0000 UTC m=+0.018076883 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:29.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:29 smithi146 podman[33757]: 2024-02-10 05:53:29.897227589 +0000 UTC m=+0.131272050 container create 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T05:53:30.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 podman[33757]: 2024-02-10 05:53:30.066129854 +0000 UTC m=+0.300174319 container init 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T05:53:30.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 podman[33757]: 2024-02-10 05:53:30.071567503 +0000 UTC m=+0.305611969 container start 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T05:53:30.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:53:30.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:53:30.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pidfile_write: ignore empty --pid-file 2024-02-10T05:53:30.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: load: jerasure load: lrc 2024-02-10T05:53:30.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:53:30.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Git sha 0 2024-02-10T05:53:30.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:53:30.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: DB SUMMARY 2024-02-10T05:53:30.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: DB Session ID: XYAKEGMT6ERSAPTN5QM3 2024-02-10T05:53:30.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:53:30.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:53:30.176 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-10T05:53:30.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-10T05:53:30.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-02-10T05:53:30.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:53:30.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:53:30.177 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:53:30.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:53:30.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:53:30.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:53:30.178 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.env: 0x5573f71e4ae0 2024-02-10T05:53:30.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:53:30.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.info_log: 0x5573f92998c0 2024-02-10T05:53:30.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:53:30.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.statistics: (nil) 2024-02-10T05:53:30.179 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.use_fsync: 0 2024-02-10T05:53:30.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:53:30.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:53:30.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:53:30.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:53:30.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:53:30.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:53:30.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:53:30.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:53:30.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:53:30.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:53:30.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:53:30.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.db_log_dir: 2024-02-10T05:53:30.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.wal_dir: 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.write_buffer_manager: 0x5573f926d400 2024-02-10T05:53:30.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.unordered_write: 0 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.row_cache: None 2024-02-10T05:53:30.184 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.wal_filter: None 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.wal_compression: 0 2024-02-10T05:53:30.186 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:53:30.187 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_open_files: -1 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:53:30.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Compression algorithms supported: 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kZSTD supported: 0 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:53:30.189 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.merge_operator: 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_filter: None 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5573f92999c0) 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2024-02-10T05:53:30.190 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 4 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x5573f929d090 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2024-02-10T05:53:30.191 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: bloomfilter 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 5 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: max_auto_readahead_size: 262144 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: prepopulate_block_cache: 0 2024-02-10T05:53:30.192 INFO:journalctl@ceph.mon.b.smithi146.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression: NoCompression 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.num_levels: 7 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:53:30.193 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:53:30.194 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:53:30.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:53:30.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:53:30.197 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.ttl: 2592000 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:53:30.198 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 81b2ecdb-4089-41a9-9674-a1b8c3acba7b 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544410101699, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544410102120, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544410, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "81b2ecdb-4089-41a9-9674-a1b8c3acba7b", "db_session_id": "XYAKEGMT6ERSAPTN5QM3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544410102199, "job": 1, "event": "recovery_finished"} 2024-02-10T05:53:30.199 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5573f938e000 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: DB pointer 0x5573f9378000 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:30.200 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:53:30.201 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache BinnedLRUCache@0x5573f929d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:53:30.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: starting mon.b rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(???) e0 preinit fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).mds e1 new map 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:53:30.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:53:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:53:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Saving service mon spec with placement smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b;count:3 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: Deploying daemon mon.b on smithi146 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:30.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/544003628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:30.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 ceph-mon[33795]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:53:30.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 bash[33757]: 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee 2024-02-10T05:53:30.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:30 smithi146 systemd[1]: Started Ceph mon.b for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:53:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/544003628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:30.674 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:53:30.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:53:30.911 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 -- 172.21.15.146:0/4003953430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d41066f0 msgr2=0x7f30bc005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 --2- 172.21.15.146:0/4003953430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d41066f0 0x7f30bc005610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f30c4005fa0 tx=0x7f30c400ffc0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 -- 172.21.15.146:0/4003953430 shutdown_connections 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 --2- 172.21.15.146:0/4003953430 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d41066f0 0x7f30bc005610 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 -- 172.21.15.146:0/4003953430 >> 172.21.15.146:0/4003953430 conn(0x7f30d40fc1c0 msgr2=0x7f30d40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 -- 172.21.15.146:0/4003953430 shutdown_connections 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.481+0000 7f30da972700 1 -- 172.21.15.146:0/4003953430 wait complete. 2024-02-10T05:53:35.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.482+0000 7f30da972700 1 Processor -- start 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: mon.a calling monitor election 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: mon.b calling monitor election 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.484 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: monmap e2: 2 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: fsmap 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: mgrmap e13: y(active, since 42s) 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: overall HEALTH_OK 2024-02-10T05:53:35.485 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:35.486 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:35.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.482+0000 7f30da972700 1 -- start start 2024-02-10T05:53:35.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30da972700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:35.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30da972700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 0x7f30d4197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30da972700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d41a0630 con 0x7f30d4197b00 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30da972700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d41a0770 con 0x7f30d41066f0 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:48764/0 (socket says 172.21.15.146:48764) 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 -- 172.21.15.146:0/2662495676 learned_addr learned my addr 172.21.15.146:0/2662495676 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d37fe700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 0x7f30d4197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 -- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 msgr2=0x7f30d41975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 -- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 msgr2=0x7f30d41975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-10T05:53:35.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d3fff700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d37fe700 1 -- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 msgr2=0x7f30d41975c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d37fe700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.483+0000 7f30d37fe700 1 -- 172.21.15.146:0/2662495676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c400f9c0 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30d37fe700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 0x7f30d4197f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f30c800b4d0 tx=0x7f30c800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f30c8011630 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f30c8011c10 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d41a0910 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f30c8010af0 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.484+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d41a0e60 con 0x7f30d4197b00 2024-02-10T05:53:35.490 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.485+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f30c801a410 con 0x7f30d4197b00 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.485+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d418f2a0 con 0x7f30d4197b00 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.485+0000 7f30d17fa700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 0x7f30c003b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.485+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f30c8011d80 con 0x7f30d4197b00 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.485+0000 7f30d3fff700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 0x7f30c003b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.486+0000 7f30d3fff700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 0x7f30c003b200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f30c400f470 tx=0x7f30c400f3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:35.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.490+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f30c8018900 con 0x7f30d4197b00 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: mon.a calling monitor election 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: mon.b calling monitor election 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: monmap e2: 2 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: fsmap 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: mgrmap e13: y(active, since 42s) 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: overall HEALTH_OK 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:35.761 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.760+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30d41a0aa0 con 0x7f30d4197b00 2024-02-10T05:53:35.762 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.761+0000 7f30d17fa700 1 -- 172.21.15.146:0/2662495676 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f30d41a0aa0 con 0x7f30d4197b00 2024-02-10T05:53:35.762 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:53:35.762 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:53:30.124776Z","created":"2024-02-10T05:51:50.247436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 msgr2=0x7f30c003b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 0x7f30c003b200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f30c400f470 tx=0x7f30c400f3c0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 msgr2=0x7f30d4197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 0x7f30d4197f80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f30c800b4d0 tx=0x7f30c800b890 comp rx=0 tx=0).stop 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 shutdown_connections 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f30c0038d40 0x7f30c003b200 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30d41066f0 0x7f30d41975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 --2- 172.21.15.146:0/2662495676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30d4197b00 0x7f30d4197f80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 >> 172.21.15.146:0/2662495676 conn(0x7f30d40fc1c0 msgr2=0x7f30d40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 shutdown_connections 2024-02-10T05:53:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:35.764+0000 7f30da972700 1 -- 172.21.15.146:0/2662495676 wait complete. 2024-02-10T05:53:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-02-10T05:53:36.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:36 smithi146 ceph-mon[33795]: Deploying daemon mon.c on smithi121 2024-02-10T05:53:36.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/2662495676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:36.336 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:36 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:36.531 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:36 smithi121 ceph-mon[30020]: Deploying daemon mon.c on smithi121 2024-02-10T05:53:36.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/2662495676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:36.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:36 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:37.362 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:53:37.362 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:53:37.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:37 smithi121 ceph-mon[30020]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:37.610 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:37 smithi146 ceph-mon[33795]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.574 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 systemd[1]: Starting Ceph mon.c for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 podman[41669]: 2024-02-10 05:53:38.574167217 +0000 UTC m=+0.111593325 container create 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 podman[41669]: 2024-02-10 05:53:38.479311642 +0000 UTC m=+0.016737760 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 podman[41669]: 2024-02-10 05:53:38.776739063 +0000 UTC m=+0.314165172 container init 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 podman[41669]: 2024-02-10 05:53:38.781962046 +0000 UTC m=+0.319388156 container start 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pidfile_write: ignore empty --pid-file 2024-02-10T05:53:38.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: load: jerasure load: lrc 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Git sha 0 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: DB SUMMARY 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: DB Session ID: PZX8YVO6YNT8B2TPLZS4 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:53:38.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:53:38.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.env: 0x563935b9aae0 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.info_log: 0x563936e978c0 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.statistics: (nil) 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.use_fsync: 0 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:53:38.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.db_log_dir: 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.wal_dir: 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:53:38.834 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.write_buffer_manager: 0x563936e6b400 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:53:38.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:53:38.836 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.unordered_write: 0 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:53:38.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.row_cache: None 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.wal_filter: None 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:53:38.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.wal_compression: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:53:38.840 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:53:38.841 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_open_files: -1 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Compression algorithms supported: 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kZSTD supported: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:53:38.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.merge_operator: 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_filter: None 2024-02-10T05:53:38.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563936e979c0) 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: index_type: 0 2024-02-10T05:53:38.844 INFO:journalctl@ceph.mon.c.smithi121.stdout: data_block_index_type: 0 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: index_shortening: 1 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: checksum: 4 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: no_block_cache: 0 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_cache: 0x563936e9b090 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_cache_options: 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: capacity : 536870912 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: num_shard_bits : 4 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: strict_capacity_limit : 0 2024-02-10T05:53:38.845 INFO:journalctl@ceph.mon.c.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_cache_compressed: (nil) 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: persistent_cache: (nil) 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_size: 4096 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_size_deviation: 10 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_restart_interval: 16 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: index_block_restart_interval: 1 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: metadata_block_size: 4096 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: partition_filters: 0 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: use_delta_encoding: 1 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: filter_policy: bloomfilter 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: whole_key_filtering: 1 2024-02-10T05:53:38.846 INFO:journalctl@ceph.mon.c.smithi121.stdout: verify_compression: 0 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: format_version: 5 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: enable_index_compression: 1 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: block_align: 0 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: max_auto_readahead_size: 262144 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: prepopulate_block_cache: 0 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:53:38.847 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression: NoCompression 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.num_levels: 7 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:53:38.848 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:53:38.849 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:53:38.850 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:53:38.851 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:53:38.852 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:53:38.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.ttl: 2592000 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:53:38.854 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:53:38.855 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9f2452a8-95d1-499a-8103-909f2249f8f4 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544418812284, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544418812799, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544418, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9f2452a8-95d1-499a-8103-909f2249f8f4", "db_session_id": "PZX8YVO6YNT8B2TPLZS4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544418812908, "job": 1, "event": "recovery_finished"} 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563936f8c000 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: DB pointer 0x563936f76000 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:53:38.856 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: ** DB Stats ** 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: ** Compaction Stats [default] ** 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:53:38.857 INFO:journalctl@ceph.mon.c.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: ** Compaction Stats [default] ** 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.858 INFO:journalctl@ceph.mon.c.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Block cache BinnedLRUCache@0x563936e9b090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-10T05:53:38.859 INFO:journalctl@ceph.mon.c.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: using public_addrv [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: starting mon.c rank -1 at public addrs [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] at bind addrs [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(???) e0 preinit fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout: e1 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:53:38.860 INFO:journalctl@ceph.mon.c.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout: legacy client fscid: -1 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout: 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout: No filesystems configured 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:53:38.861 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3742251514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:53:38.862 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:38.863 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Saving service mon spec with placement smithi121:172.21.15.121=a;smithi121:[v2:172.21.15.121:3301,v1:172.21.15.121:6790]=c;smithi146:172.21.15.146=b;count:3 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:38.864 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Deploying daemon mon.b on smithi146 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/544003628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.a calling monitor election 2024-02-10T05:53:38.865 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.b calling monitor election 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: monmap e2: 2 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: fsmap 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mgrmap e13: y(active, since 42s) 2024-02-10T05:53:38.866 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: overall HEALTH_OK 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: Deploying daemon mon.c on smithi121 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/2662495676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:38.867 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 ceph-mon[41707]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:53:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 bash[41669]: 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed 2024-02-10T05:53:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:38 smithi121 systemd[1]: Started Ceph mon.c for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:53:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: mon.a calling monitor election 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: mon.b calling monitor election 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: mon.c calling monitor election 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:53:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: monmap e3: 3 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: fsmap 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: mgrmap e13: y(active, since 51s) 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: overall HEALTH_OK 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: mon.a calling monitor election 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: mon.b calling monitor election 2024-02-10T05:53:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: mon.c calling monitor election 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: monmap e3: 3 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: fsmap 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:44.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: mgrmap e13: y(active, since 51s) 2024-02-10T05:53:44.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: overall HEALTH_OK 2024-02-10T05:53:44.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: mon.a calling monitor election 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: mon.b calling monitor election 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: mon.c calling monitor election 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: monmap e3: 3 mons at {a=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: fsmap 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: mgrmap e13: y(active, since 51s) 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: overall HEALTH_OK 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:44.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:45.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.154+0000 7f02a97c0700 1 -- 172.21.15.146:0/3659341775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028c0062a0 msgr2=0x7f028c008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.154+0000 7f02a97c0700 1 --2- 172.21.15.146:0/3659341775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028c0062a0 0x7f028c008710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02a41028c0 tx=0x7f029400fe90 comp rx=0 tx=0).stop 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 -- 172.21.15.146:0/3659341775 shutdown_connections 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 --2- 172.21.15.146:0/3659341775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f028c0062a0 0x7f028c008710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 -- 172.21.15.146:0/3659341775 >> 172.21.15.146:0/3659341775 conn(0x7f02a40fd2f0 msgr2=0x7f02a40ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 -- 172.21.15.146:0/3659341775 shutdown_connections 2024-02-10T05:53:45.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 -- 172.21.15.146:0/3659341775 wait complete. 2024-02-10T05:53:45.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:44 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:45.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:44 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:45.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:44 smithi146 ceph-mon[33795]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:45.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:44 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.155+0000 7f02a97c0700 1 Processor -- start 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 -- start start 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f028c0062a0 0x7f02a419c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a4196d00 0x7f02a41971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a419d1f0 con 0x7f02a4196d00 2024-02-10T05:53:45.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a41a34a0 con 0x7f02a419cb80 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a97c0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a41a3610 con 0x7f028c0062a0 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a27fc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a27fc700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34486/0 (socket says 172.21.15.146:34486) 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a27fc700 1 -- 172.21.15.146:0/1404033021 learned_addr learned my addr 172.21.15.146:0/1404033021 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a37fe700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a4196d00 0x7f02a41971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a2ffd700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f028c0062a0 0x7f02a419c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.156+0000 7f02a27fc700 1 -- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f028c0062a0 msgr2=0x7f02a419c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a27fc700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f028c0062a0 0x7f02a419c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a27fc700 1 -- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a4196d00 msgr2=0x7f02a41971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a27fc700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a4196d00 0x7f02a41971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a27fc700 1 -- 172.21.15.146:0/1404033021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029400f8e0 con 0x7f02a419cb80 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a27fc700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f029800b780 tx=0x7f029800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:45.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0298004230 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0298004390 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a41a38f0 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02980119d0 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.157+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a41a3df0 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.158+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f0298011b30 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.158+0000 7f028bfff700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 0x7f029003b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.158+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a4102f40 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.158+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f029804a920 con 0x7f02a419cb80 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.158+0000 7f02a2ffd700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 0x7f029003b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:45.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.159+0000 7f02a2ffd700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 0x7f029003b8d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0294010010 tx=0x7f0294005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:45.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.163+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0298018da0 con 0x7f02a419cb80 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[41707]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[30020]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:44 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:53:45.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.437+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02a4197cf0 con 0x7f02a419cb80 2024-02-10T05:53:45.438 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.437+0000 7f028bfff700 1 -- 172.21.15.146:0/1404033021 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f0298016030 con 0x7f02a419cb80 2024-02-10T05:53:45.438 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-02-10T05:53:45.438 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:53:38.834327Z","created":"2024-02-10T05:51:50.247436Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3301","nonce":0},{"type":"v1","addr":"172.21.15.121:6790","nonce":0}]},"addr":"172.21.15.121:6790/0","public_addr":"172.21.15.121:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 msgr2=0x7f029003b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 0x7f029003b8d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0294010010 tx=0x7f0294005da0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 msgr2=0x7f02a41967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f029800b780 tx=0x7f029800ba90 comp rx=0 tx=0).stop 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 shutdown_connections 2024-02-10T05:53:45.440 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a4196d00 0x7f02a41971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0290039410 0x7f029003b8d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f028c0062a0 0x7f02a419c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 --2- 172.21.15.146:0/1404033021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a419cb80 0x7f02a41967c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.439+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 >> 172.21.15.146:0/1404033021 conn(0x7f02a40fd2f0 msgr2=0x7f02a4105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.440+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 shutdown_connections 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:45.440+0000 7f02a97c0700 1 -- 172.21.15.146:0/1404033021 wait complete. 2024-02-10T05:53:45.441 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 3 2024-02-10T05:53:46.144 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T05:53:46.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T05:53:46.391 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/1404033021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/1404033021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.506 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:46.507 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/1404033021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.508 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.509 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:46.510 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:46.510 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:46.510 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:47.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[30020]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:53:47.551 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[30020]: Reconfiguring daemon mon.a on smithi121 2024-02-10T05:53:47.551 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[30020]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:47.551 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[41707]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:53:47.551 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[41707]: Reconfiguring daemon mon.a on smithi121 2024-02-10T05:53:47.551 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:47 smithi121 ceph-mon[41707]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:47 smithi146 ceph-mon[33795]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:53:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:47 smithi146 ceph-mon[33795]: Reconfiguring daemon mon.a on smithi121 2024-02-10T05:53:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:47 smithi146 ceph-mon[33795]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:48.942 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:48.943 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:48.944 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:48.944 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:48 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:49.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:48 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 -- 172.21.15.121:0/517226444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb41015c0 msgr2=0x7f1bb410daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 --2- 172.21.15.121:0/517226444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb41015c0 0x7f1bb410daa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009f90 tx=0x7f1bac00b330 comp rx=0 tx=0).stop 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 -- 172.21.15.121:0/517226444 shutdown_connections 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 --2- 172.21.15.121:0/517226444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb41015c0 0x7f1bb410daa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 --2- 172.21.15.121:0/517226444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb4106c60 0x7f1bb4107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 --2- 172.21.15.121:0/517226444 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4100c20 0x7f1bb4101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 -- 172.21.15.121:0/517226444 >> 172.21.15.121:0/517226444 conn(0x7f1bb4077b50 msgr2=0x7f1bb4077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 -- 172.21.15.121:0/517226444 shutdown_connections 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.183+0000 7f1bb941e700 1 -- 172.21.15.121:0/517226444 wait complete. 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 Processor -- start 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 -- start start 2024-02-10T05:53:49.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 0x7f1bb419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 0x7f1bb41976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4197e70 con 0x7f1bb4100c20 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4197fe0 con 0x7f1bb41015c0 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb941e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb41981a0 con 0x7f1bb4106c60 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56150/0 (socket says 172.21.15.121:56150) 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.184+0000 7f1bb2ffd700 1 -- 172.21.15.121:0/93506005 learned_addr learned my addr 172.21.15.121:0/93506005 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb37fe700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 0x7f1bb41976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb27fc700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 0x7f1bb419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 -- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 msgr2=0x7f1bb41976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 0x7f1bb41976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 -- 172.21.15.121:0/93506005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 msgr2=0x7f1bb419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 0x7f1bb419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 -- 172.21.15.121:0/93506005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80095f0 con 0x7f1bb4100c20 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb37fe700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 0x7f1bb41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb27fc700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 0x7f1bb419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb2ffd700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d950 tx=0x7f1ba400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4009940 con 0x7f1bb4100c20 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1ba4010460 con 0x7f1bb4100c20 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400f620 con 0x7f1bb4100c20 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac009c40 con 0x7f1bb4100c20 2024-02-10T05:53:49.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.185+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41a4570 con 0x7f1bb4100c20 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.186+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f1ba400f780 con 0x7f1bb4100c20 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.186+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb4102310 con 0x7f1bb4100c20 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.186+0000 7f1b9bfff700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 0x7f1b9c03b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.186+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ba404bc80 con 0x7f1bb4100c20 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.186+0000 7f1bb27fc700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 0x7f1b9c03b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:49.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.187+0000 7f1bb27fc700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 0x7f1b9c03b680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4198e80 tx=0x7f1ba8009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:49.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.192+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1ba40226c0 con 0x7f1bb4100c20 2024-02-10T05:53:49.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.394+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1bb4198bf0 con 0x7f1bb4100c20 2024-02-10T05:53:49.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.394+0000 7f1b9bfff700 1 -- 172.21.15.121:0/93506005 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f1ba4022300 con 0x7f1bb4100c20 2024-02-10T05:53:49.395 INFO:teuthology.orchestra.run.smithi121.stdout:# minimal ceph.conf for 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:49.395 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-02-10T05:53:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:53:49.395 INFO:teuthology.orchestra.run.smithi121.stdout: mon_host = [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 msgr2=0x7f1b9c03b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 0x7f1b9c03b680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4198e80 tx=0x7f1ba8009310 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 msgr2=0x7f1bb419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400d950 tx=0x7f1ba400dc60 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 shutdown_connections 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b9c0391c0 0x7f1b9c03b680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bb4106c60 0x7f1bb41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb4100c20 0x7f1bb419d5f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.396+0000 7f1bb941e700 1 --2- 172.21.15.121:0/93506005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bb41015c0 0x7f1bb419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.397+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 >> 172.21.15.121:0/93506005 conn(0x7f1bb4077b50 msgr2=0x7f1bb40fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:49.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.397+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 shutdown_connections 2024-02-10T05:53:49.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:49.397+0000 7f1bb941e700 1 -- 172.21.15.121:0/93506005 wait complete. 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[41707]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[41707]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[41707]: Reconfiguring daemon mon.c on smithi121 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/93506005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[30020]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[30020]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[30020]: Reconfiguring daemon mon.c on smithi121 2024-02-10T05:53:50.035 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/93506005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.062 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T05:53:50.062 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:50.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:53:50.091 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:50.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:50.159 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:53:50.159 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:53:50.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:49 smithi146 ceph-mon[33795]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:50.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:49 smithi146 ceph-mon[33795]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:53:50.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:49 smithi146 ceph-mon[33795]: Reconfiguring daemon mon.c on smithi121 2024-02-10T05:53:50.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/93506005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.194 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:53:50.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:53:50.265 INFO:tasks.cephadm:Adding mgr.y on smithi121 2024-02-10T05:53:50.265 INFO:tasks.cephadm:Adding mgr.x on smithi146 2024-02-10T05:53:50.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi121=y;smithi146=x' 2024-02-10T05:53:50.544 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:51.885 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:51.886 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:53:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:52.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.140+0000 7fe2a45b2700 1 -- 172.21.15.146:0/3374547335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 msgr2=0x7fe29c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:52.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.140+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/3374547335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c100ba0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009910 tx=0x7fe28c009c20 comp rx=0 tx=0).stop 2024-02-10T05:53:52.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.140+0000 7fe2a45b2700 1 -- 172.21.15.146:0/3374547335 shutdown_connections 2024-02-10T05:53:52.141 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.140+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/3374547335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c100ba0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.140+0000 7fe2a45b2700 1 -- 172.21.15.146:0/3374547335 >> 172.21.15.146:0/3374547335 conn(0x7fe29c0fbd20 msgr2=0x7fe29c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 -- 172.21.15.146:0/3374547335 shutdown_connections 2024-02-10T05:53:52.142 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 -- 172.21.15.146:0/3374547335 wait complete. 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 Processor -- start 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 -- start start 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.141+0000 7fe2a45b2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c195cc0 0x7fe29c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a45b2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c19d5e0 con 0x7fe29c100780 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a45b2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c19d750 con 0x7fe29c195840 2024-02-10T05:53:52.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a45b2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c19d910 con 0x7fe29c195cc0 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34498/0 (socket says 172.21.15.146:34498) 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 -- 172.21.15.146:0/76941070 learned_addr learned my addr 172.21.15.146:0/76941070 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a234e700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a2b4f700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c195cc0 0x7fe29c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 -- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c195cc0 msgr2=0x7fe29c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c195cc0 0x7fe29c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 -- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 msgr2=0x7fe29c195300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c195300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 -- 172.21.15.146:0/76941070 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c0095f0 con 0x7fe29c195840 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a1b4d700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe29800b780 tx=0x7fe29800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298004230 con 0x7fe29c195840 2024-02-10T05:53:52.145 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c19dbf0 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe298004390 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.142+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe298011a40 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.143+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c19e1c0 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.143+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fe298011ba0 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.143+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c063350 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.144+0000 7fe2937fe700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 0x7fe28803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.144+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe29804a880 con 0x7fe29c195840 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.144+0000 7fe2a234e700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 0x7fe28803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:52.146 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.144+0000 7fe2a234e700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 0x7fe28803b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009fd0 tx=0x7fe28c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:52.149 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.149+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe2980036a0 con 0x7fe29c195840 2024-02-10T05:53:52.361 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.360+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi146=x", "target": ["mon-mgr", ""]}) v1 -- 0x7fe29c18ea80 con 0x7fe288039410 2024-02-10T05:53:52.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.367+0000 7fe2937fe700 1 -- 172.21.15.146:0/76941070 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe29c18ea80 con 0x7fe288039410 2024-02-10T05:53:52.367 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mgr update... 2024-02-10T05:53:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 msgr2=0x7fe28803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:52.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 0x7fe28803b8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe28c009fd0 tx=0x7fe28c0058e0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 msgr2=0x7fe29c19a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe29800b780 tx=0x7fe29800ba90 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 shutdown_connections 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fe288039410 0x7fe28803b8d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c195cc0 0x7fe29c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c100780 0x7fe29c195300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 --2- 172.21.15.146:0/76941070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe29c195840 0x7fe29c19a8b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 >> 172.21.15.146:0/76941070 conn(0x7fe29c0fbd20 msgr2=0x7fe29c0fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 shutdown_connections 2024-02-10T05:53:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:53:52.369+0000 7fe2a45b2700 1 -- 172.21.15.146:0/76941070 wait complete. 2024-02-10T05:53:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:52 smithi146 ceph-mon[33795]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:53:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:52 smithi146 ceph-mon[33795]: Reconfiguring daemon mon.b on smithi146 2024-02-10T05:53:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:52.966 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T05:53:52.970 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T05:53:52.970 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:53:52.970 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:53:52.989 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:53:52.990 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-02-10T05:53:53.048 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:53:53.048 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.048 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-10 05:53:38.302383467 +0000 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-10 05:48:08.029389796 +0000 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-10 05:48:08.029389796 +0000 2024-02-10T05:53:53.049 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-02-10 05:48:08.029389796 +0000 2024-02-10T05:53:53.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:53:53.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[30020]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:53:53.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[30020]: Reconfiguring daemon mon.b on smithi146 2024-02-10T05:53:53.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:53.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[41707]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:53:53.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[41707]: Reconfiguring daemon mon.b on smithi146 2024-02-10T05:53:53.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:53.110 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-10T05:53:53.110 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-10T05:53:53.111 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000240181 s, 2.1 MB/s 2024-02-10T05:53:53.112 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:53:53.170 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-10 05:53:38.303383454 +0000 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-10 05:48:08.369385389 +0000 2024-02-10T05:53:53.227 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-10 05:48:08.369385389 +0000 2024-02-10T05:53:53.228 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-02-10 05:48:08.369385389 +0000 2024-02-10T05:53:53.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:53:53.295 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-10T05:53:53.295 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-10T05:53:53.295 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000202658 s, 2.5 MB/s 2024-02-10T05:53:53.297 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:53:53.354 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-02-10T05:53:53.411 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:53:53.411 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.411 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-10 05:53:38.303383454 +0000 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-10 05:48:08.716380892 +0000 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-10 05:48:08.716380892 +0000 2024-02-10T05:53:53.412 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-02-10 05:48:08.716380892 +0000 2024-02-10T05:53:53.412 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:53:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-10T05:53:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-10T05:53:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000224238 s, 2.3 MB/s 2024-02-10T05:53:53.480 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:53:53.536 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-02-10T05:53:53.592 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-02-10 05:53:38.303383454 +0000 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-02-10 05:48:09.065376369 +0000 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-02-10 05:48:09.065376369 +0000 2024-02-10T05:53:53.593 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-02-10 05:48:09.065376369 +0000 2024-02-10T05:53:53.593 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:53:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-02-10T05:53:53.659 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-02-10T05:53:53.660 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00020938 s, 2.4 MB/s 2024-02-10T05:53:53.661 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:53:53.718 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:53:53.718 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:53:53.735 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:53:53.735 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-02-10T05:53:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:53:53.791 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.791 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-02-10 05:53:29.058868571 +0000 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-02-10 05:48:08.031297421 +0000 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-02-10 05:48:08.031297421 +0000 2024-02-10T05:53:53.792 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-02-10 05:48:08.031297421 +0000 2024-02-10T05:53:53.792 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:53:53.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:53 smithi146 ceph-mon[33795]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:53.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:53 smithi146 ceph-mon[33795]: Saving service mgr spec with placement smithi121=y;smithi146=x;count:2 2024-02-10T05:53:53.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:53 smithi146 ceph-mon[33795]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:53.857 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-02-10T05:53:53.857 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-02-10T05:53:53.857 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000219856 s, 2.3 MB/s 2024-02-10T05:53:53.858 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:53:53.913 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-02-10 05:53:29.058868571 +0000 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-02-10 05:48:08.374292179 +0000 2024-02-10T05:53:53.969 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-02-10 05:48:08.374292179 +0000 2024-02-10T05:53:53.970 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-02-10 05:48:08.374292179 +0000 2024-02-10T05:53:53.970 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:53:54.036 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-02-10T05:53:54.036 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-02-10T05:53:54.036 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000225434 s, 2.3 MB/s 2024-02-10T05:53:54.037 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[30020]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[30020]: Saving service mgr spec with placement smithi121=y;smithi146=x;count:2 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[30020]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[41707]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi121=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[41707]: Saving service mgr spec with placement smithi121=y;smithi146=x;count:2 2024-02-10T05:53:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:53 smithi121 ceph-mon[41707]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:54.095 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-02-10T05:53:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:53:54.150 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-02-10 05:53:29.059868556 +0000 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-02-10 05:48:08.715286967 +0000 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-02-10 05:48:08.715286967 +0000 2024-02-10T05:53:54.151 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-02-10 05:48:08.715286967 +0000 2024-02-10T05:53:54.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:53:54.216 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-02-10T05:53:54.216 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-02-10T05:53:54.216 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000200767 s, 2.6 MB/s 2024-02-10T05:53:54.217 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:53:54.273 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-02-10T05:53:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:53:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:53:54.329 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-02-10 05:53:29.059868556 +0000 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-02-10 05:48:09.074281481 +0000 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-02-10 05:48:09.074281481 +0000 2024-02-10T05:53:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-02-10 05:48:09.074281481 +0000 2024-02-10T05:53:54.330 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:53:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-02-10T05:53:54.395 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-02-10T05:53:54.396 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000215065 s, 2.4 MB/s 2024-02-10T05:53:54.397 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:53:54.452 INFO:tasks.cephadm:Deploying osd.0 on smithi121 with /dev/vg_nvme/lv_4... 2024-02-10T05:53:54.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:53:54.698 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[41707]: Deploying daemon mgr.x on smithi146 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:55.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:53:55.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:55.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:54 smithi121 ceph-mon[30020]: Deploying daemon mgr.x on smithi146 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:55.159 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:53:55.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:55.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:54 smithi146 ceph-mon[33795]: Deploying daemon mgr.x on smithi146 2024-02-10T05:53:56.094 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:55 smithi146 ceph-mon[33795]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:56.094 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:55 smithi146 systemd[1]: Starting Ceph mgr.x for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:53:56.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:55 smithi121 ceph-mon[30020]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:56.103 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:55 smithi121 ceph-mon[41707]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:56.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 podman[35853]: 2024-02-10 05:53:55.996900751 +0000 UTC m=+0.018269485 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:53:56.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 podman[35853]: 2024-02-10 05:53:56.095012348 +0000 UTC m=+0.116381079 container create 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T05:53:56.372 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 podman[35853]: 2024-02-10 05:53:56.37237758 +0000 UTC m=+0.393746304 container init 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T05:53:56.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 podman[35853]: 2024-02-10 05:53:56.377933061 +0000 UTC m=+0.399301789 container start 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:53:56.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 ceph-mgr[35892]: -- 172.21.15.146:0/2282991637 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e360a21a0 con 0x557e36079000 2024-02-10T05:53:56.672 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 bash[35853]: 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 2024-02-10T05:53:56.672 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 systemd[1]: Started Ceph mgr.x for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013191 s, 795 MB/s 2024-02-10T05:53:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-10T05:53:57.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-02-10T05:53:57.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:56.937+0000 7f499a45e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:53:57.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:57.034+0000 7f499a45e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:53:57.528 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:53:57.783 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.783 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.784 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.785 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:57.785 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:57.819 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:57.724+0000 7f499a45e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:53:57.819 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:57.819+0000 7f499a45e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:57.820 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:58.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:57.965+0000 7f499a45e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:53:58.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.067+0000 7f499a45e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:53:58.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.153+0000 7f499a45e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:53:58.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.414+0000 7f499a45e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:53:58.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.524+0000 7f499a45e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:53:58.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.611+0000 7f499a45e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.994 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:58.995 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:59.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.707+0000 7f499a45e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:58.886+0000 7f499a45e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:59.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:53:59.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:53:59.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:53:59.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- 172.21.15.121:0/688987854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90100cc0 msgr2=0x7f8c901010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 --2- 172.21.15.121:0/688987854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90100cc0 0x7f8c901010e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009990 tx=0x7f8c80009ca0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- 172.21.15.121:0/688987854 shutdown_connections 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 --2- 172.21.15.121:0/688987854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90102860 0x7f8c90109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 --2- 172.21.15.121:0/688987854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90100cc0 0x7f8c901010e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 --2- 172.21.15.121:0/688987854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 0x7f8c90102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- 172.21.15.121:0/688987854 >> 172.21.15.121:0/688987854 conn(0x7f8c900fc280 msgr2=0x7f8c900fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- 172.21.15.121:0/688987854 shutdown_connections 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- 172.21.15.121:0/688987854 wait complete. 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 Processor -- start 2024-02-10T05:53:59.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.751+0000 7f8c96a13700 1 -- start start 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 0x7f8c90109c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 0x7f8c9010a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9010a8e0 con 0x7f8c90100cc0 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9010aa50 con 0x7f8c90101ec0 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c96a13700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c901a12e0 con 0x7f8c90102860 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c94fb0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 0x7f8c9010a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45998/0 (socket says 172.21.15.121:45998) 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 -- 172.21.15.121:0/1882274962 learned_addr learned my addr 172.21.15.121:0/1882274962 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8f7fe700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 0x7f8c90109c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:59.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 -- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 msgr2=0x7f8c9010a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 0x7f8c9010a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 -- 172.21.15.121:0/1882274962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 msgr2=0x7f8c90109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 0x7f8c90109c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 -- 172.21.15.121:0/1882274962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c88009c40 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c94fb0700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 0x7f8c9010a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.752+0000 7f8c8ffff700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c80000c00 tx=0x7f8c80003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.753+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8001d070 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.753+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8c8000be60 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.753+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c800095f0 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.753+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80022de0 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.753+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c901a17e0 con 0x7f8c90100cc0 2024-02-10T05:53:59.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 45989+0+0 (secure 0 0 0) 0x7f8c8000f460 con 0x7f8c90100cc0 2024-02-10T05:53:59.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c8d7fa700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 0x7f8c7803b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:53:59.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c901081a0 con 0x7f8c90100cc0 2024-02-10T05:53:59.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8c8004d2f0 con 0x7f8c90100cc0 2024-02-10T05:53:59.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c8f7fe700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 0x7f8c7803b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:53:59.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.754+0000 7f8c8f7fe700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 0x7f8c7803b680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c84007c60 tx=0x7f8c840073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:53:59.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.759+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8c80016580 con 0x7f8c90100cc0 2024-02-10T05:53:59.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:53:59.967+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c9010cdf0 con 0x7f8c780391c0 2024-02-10T05:53:59.968 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[41707]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:53:59.968 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[41707]: Reconfiguring daemon mgr.y on smithi121 2024-02-10T05:53:59.968 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[41707]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:59.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[30020]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:53:59.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[30020]: Reconfiguring daemon mgr.y on smithi121 2024-02-10T05:53:59.969 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:53:59 smithi121 ceph-mon[30020]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:59.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:59 smithi146 ceph-mon[33795]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:53:59.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:59 smithi146 ceph-mon[33795]: Reconfiguring daemon mgr.y on smithi121 2024-02-10T05:53:59.996 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:53:59 smithi146 ceph-mon[33795]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:53:59.996 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:59.864+0000 7f499a45e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:53:59.997 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:53:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:53:59.995+0000 7f499a45e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:54:00.614 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:00.331+0000 7f499a45e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:54:00.614 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:00.422+0000 7f499a45e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:54:00.614 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:00.613+0000 7f499a45e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:54:00.834 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:00.834 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.835 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.836 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:00.836 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.836 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:00.836 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.921 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:00.797+0000 7f499a45e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:54:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:01.341 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:01.078+0000 7f499a45e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:54:01.341 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:01.340+0000 7f499a45e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:54:01.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:01.574+0000 7f499a45e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:54:02.001 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:01 smithi121 ceph-mon[30020]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:02.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:01 smithi121 ceph-mon[30020]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:02.002 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:01 smithi121 ceph-mon[41707]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:02.002 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:01 smithi121 ceph-mon[41707]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:01 smithi146 ceph-mon[33795]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:01 smithi146 ceph-mon[33795]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:02.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:02.379+0000 7f499a45e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:02.954+0000 7f499a45e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:03.053+0000 7f499a45e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]: dispatch 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]': finished 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-mon[33795]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:54:03.054 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:02 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]: dispatch 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]': finished 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[30020]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]: dispatch 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2376297425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9e9f036-b22d-4b48-a265-3413e9eda4f0"}]': finished 2024-02-10T05:54:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[41707]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:54:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:02 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:03.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:54:03.276+0000 7f499a45e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:54:03.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:03.892+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 14) v1 ==== 91351+0+0 (secure 0 0 0) 0x7f8c8002c440 con 0x7f8c90100cc0 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3259393672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: Standby manager daemon x started 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:54:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3259393672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: Standby manager daemon x started 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:54:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:03 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:54:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3259393672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: Standby manager daemon x started 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:54:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:03 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/2431580278' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:54:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:04 smithi146 ceph-mon[33795]: mgrmap e14: y(active, since 71s), standbys: x 2024-02-10T05:54:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:54:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:04 smithi146 ceph-mon[33795]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[41707]: mgrmap e14: y(active, since 71s), standbys: x 2024-02-10T05:54:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:54:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[41707]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[30020]: mgrmap e14: y(active, since 71s), standbys: x 2024-02-10T05:54:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:54:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:04 smithi121 ceph-mon[30020]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:08.050 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:07 smithi121 ceph-mon[30020]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:08.050 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:07 smithi121 ceph-mon[41707]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:07 smithi146 ceph-mon[33795]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:09.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:08 smithi121 ceph-mon[30020]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:09.003 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:08 smithi121 ceph-mon[41707]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:09.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:08 smithi146 ceph-mon[33795]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:09 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:54:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:09 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:10.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:09 smithi146 ceph-mon[33795]: Deploying daemon osd.0 on smithi121 2024-02-10T05:54:10.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:54:10.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:10.326 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[30020]: Deploying daemon osd.0 on smithi121 2024-02-10T05:54:10.326 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:54:10.326 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:10.327 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:09 smithi121 ceph-mon[41707]: Deploying daemon osd.0 on smithi121 2024-02-10T05:54:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:10 smithi146 ceph-mon[33795]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:10 smithi121 ceph-mon[30020]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:11.275 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:10 smithi121 ceph-mon[41707]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:13 smithi121 ceph-mon[30020]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:13.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:13 smithi121 ceph-mon[41707]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:14.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:13 smithi146 ceph-mon[33795]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[30020]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:15 smithi121 ceph-mon[41707]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:15.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:15 smithi146 ceph-mon[33795]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:18.044 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[30020]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.045 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.046 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:17 smithi121 ceph-mon[41707]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.120+0000 7f8c8d7fa700 1 -- 172.21.15.121:0/1882274962 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c9010cdf0 con 0x7f8c780391c0 2024-02-10T05:54:18.121 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 0 on host 'smithi121' 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 msgr2=0x7f8c7803b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 0x7f8c7803b680 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c84007c60 tx=0x7f8c840073d0 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 msgr2=0x7f8c9010b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c80000c00 tx=0x7f8c80003920 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 shutdown_connections 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8c780391c0 0x7f8c7803b680 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c90102860 0x7f8c9010a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c90100cc0 0x7f8c9010b5d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 --2- 172.21.15.121:0/1882274962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c90101ec0 0x7f8c90109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:18.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 >> 172.21.15.121:0/1882274962 conn(0x7f8c900fc280 msgr2=0x7f8c901052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:54:18.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 shutdown_connections 2024-02-10T05:54:18.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:18.123+0000 7f8c96a13700 1 -- 172.21.15.121:0/1882274962 wait complete. 2024-02-10T05:54:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:17 smithi146 ceph-mon[33795]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:54:18.743 DEBUG:teuthology.orchestra.run.smithi121:osd.0> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T05:54:18.746 INFO:tasks.cephadm:Deploying osd.1 on smithi121 with /dev/vg_nvme/lv_3... 2024-02-10T05:54:18.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:54:19.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[30020]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:54:19.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:54:19.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:19.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-mon[41707]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:19.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:19 smithi146 ceph-mon[33795]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:19.502 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:20.075 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T05:54:19.690+0000 7f117bdaa700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T05:54:20.075 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:54:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T05:54:19.694+0000 7f1173394700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:54:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[30020]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[30020]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:54:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[30020]: from='osd.0 ' entity='osd.0' 2024-02-10T05:54:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[41707]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[41707]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:54:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:20 smithi121 ceph-mon[41707]: from='osd.0 ' entity='osd.0' 2024-02-10T05:54:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:20 smithi146 ceph-mon[33795]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:20 smithi146 ceph-mon[33795]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:54:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:20 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:20 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:20 smithi146 ceph-mon[33795]: from='osd.0 ' entity='osd.0' 2024-02-10T05:54:21.541 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:54:21.541 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] boot 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] boot 2024-02-10T05:54:21.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:54:21.543 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: osd.0 [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] boot 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:54:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0174106 s, 602 MB/s 2024-02-10T05:54:22.391 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-10T05:54:23.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi121 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 16064M 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[30020]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi121 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 16064M 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:23 smithi121 ceph-mon[41707]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:23.332 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi121 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 16064M 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:23 smithi146 ceph-mon[33795]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.987+0000 7fede723e700 1 -- 172.21.15.121:0/1539158174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 msgr2=0x7fede0102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.987+0000 7fede723e700 1 --2- 172.21.15.121:0/1539158174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0102190 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009990 tx=0x7fedd4009ca0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 -- 172.21.15.121:0/1539158174 shutdown_connections 2024-02-10T05:54:24.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- 172.21.15.121:0/1539158174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0103910 0x7fede010a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- 172.21.15.121:0/1539158174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0102190 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- 172.21.15.121:0/1539158174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0102f70 0x7fede01033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 -- 172.21.15.121:0/1539158174 >> 172.21.15.121:0/1539158174 conn(0x7fede00fd330 msgr2=0x7fede00ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 -- 172.21.15.121:0/1539158174 shutdown_connections 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 -- 172.21.15.121:0/1539158174 wait complete. 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 Processor -- start 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 -- start start 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.988+0000 7fede723e700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 0x7fede0074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede723e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0074dd0 con 0x7fede0102f70 2024-02-10T05:54:24.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede723e700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede0074f40 con 0x7fede0103910 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede723e700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede01a1330 con 0x7fede0101d70 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7feddffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:34970/0 (socket says 172.21.15.121:34970) 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7feddffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48030/0 (socket says 172.21.15.121:48030) 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 -- 172.21.15.121:0/3570686496 learned_addr learned my addr 172.21.15.121:0/3570686496 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede57db700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 0x7fede0074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 -- 172.21.15.121:0/3570686496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 msgr2=0x7fede0074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 0x7fede0074650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 -- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 msgr2=0x7fede0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 -- 172.21.15.121:0/3570686496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8009c40 con 0x7fede0101d70 2024-02-10T05:54:24.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede57db700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 0x7fede0074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7feddffff700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede4fda700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fedd40098e0 tx=0x7fedd4003980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd401d070 con 0x7fede0101d70 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd40095f0 con 0x7fede0101d70 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fedd400bde0 con 0x7fede0101d70 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400fd80 con 0x7fede0101d70 2024-02-10T05:54:24.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.989+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede01a1890 con 0x7fede0101d70 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.991+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede0108950 con 0x7fede0101d70 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.991+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 14) v1 ==== 91351+0+0 (secure 0 0 0) 0x7fedd402b440 con 0x7fede0101d70 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.991+0000 7fedddffb700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 0x7fedd006fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.991+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fedd408d4a0 con 0x7fede0101d70 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.991+0000 7feddffff700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 0x7fedd006fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:24.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.992+0000 7feddffff700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 0x7fedd006fe00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedcc005950 tx=0x7fedcc00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:54:24.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:24.996+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fedd40580e0 con 0x7fede0101d70 2024-02-10T05:54:25.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:25.198+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fede010cc80 con 0x7fedd006d940 2024-02-10T05:54:25.996 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[30020]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:25.996 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:25.996 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:25.996 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:25.997 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[41707]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:25.997 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:25.997 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:25.997 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:25 smithi146 ceph-mon[33795]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:27.005 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:26 smithi121 ceph-mon[30020]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:27.006 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:26 smithi121 ceph-mon[41707]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:26 smithi146 ceph-mon[33795]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:27 smithi121 ceph-mon[41707]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:27 smithi121 ceph-mon[30020]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:27 smithi146 ceph-mon[33795]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]: dispatch 2024-02-10T05:54:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]': finished 2024-02-10T05:54:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[41707]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:54:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753420031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]: dispatch 2024-02-10T05:54:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]': finished 2024-02-10T05:54:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[30020]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:54:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753420031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:29.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]: dispatch 2024-02-10T05:54:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3411529003' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f733eda4-d0b0-4b09-84f2-5ec618b890db"}]': finished 2024-02-10T05:54:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:28 smithi146 ceph-mon[33795]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:54:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753420031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:30.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:29 smithi121 ceph-mon[41707]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:30.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:29 smithi121 ceph-mon[30020]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:30.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:29 smithi146 ceph-mon[33795]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:32.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:31 smithi121 ceph-mon[30020]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:31 smithi121 ceph-mon[41707]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:32.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:31 smithi146 ceph-mon[33795]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:34.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:33 smithi121 ceph-mon[30020]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:34.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:33 smithi121 ceph-mon[41707]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:34.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:33 smithi146 ceph-mon[33795]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:35.063 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:34 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:54:35.063 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:34 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:35.063 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:34 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:54:35.063 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:34 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:35.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:34 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:54:35.164 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:34 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:36.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:35 smithi121 ceph-mon[30020]: Deploying daemon osd.1 on smithi121 2024-02-10T05:54:36.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:35 smithi121 ceph-mon[30020]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:36.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:35 smithi121 ceph-mon[41707]: Deploying daemon osd.1 on smithi121 2024-02-10T05:54:36.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:35 smithi121 ceph-mon[41707]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:35 smithi146 ceph-mon[33795]: Deploying daemon osd.1 on smithi121 2024-02-10T05:54:36.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:35 smithi146 ceph-mon[33795]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:37.967 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:37 smithi121 ceph-mon[30020]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:37.967 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:37 smithi121 ceph-mon[41707]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:38.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:37 smithi146 ceph-mon[33795]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:39.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:38 smithi146 ceph-mon[33795]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:39.256 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:38 smithi121 ceph-mon[30020]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:39.256 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:38 smithi121 ceph-mon[41707]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:41.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:41.919 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:41 smithi121 ceph-mon[30020]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:41.919 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:41 smithi121 ceph-mon[41707]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:42.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:41 smithi146 ceph-mon[33795]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:43.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.156+0000 7fedddffb700 1 -- 172.21.15.121:0/3570686496 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fede010cc80 con 0x7fedd006d940 2024-02-10T05:54:43.157 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 1 on host 'smithi121' 2024-02-10T05:54:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 msgr2=0x7fedd006fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:43.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 0x7fedd006fe00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fedcc005950 tx=0x7fedcc00a300 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 msgr2=0x7fede0075ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fedd40098e0 tx=0x7fedd4003980 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 shutdown_connections 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fede0103910 0x7fede0074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fedd006d940 0x7fedd006fe00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fede0101d70 0x7fede0075ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 --2- 172.21.15.121:0/3570686496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0102f70 0x7fede0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 >> 172.21.15.121:0/3570686496 conn(0x7fede00fd330 msgr2=0x7fede0100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 shutdown_connections 2024-02-10T05:54:43.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:43.159+0000 7fede723e700 1 -- 172.21.15.121:0/3570686496 wait complete. 2024-02-10T05:54:43.723 DEBUG:teuthology.orchestra.run.smithi121:osd.1> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T05:54:43.725 INFO:tasks.cephadm:Deploying osd.2 on smithi121 with /dev/vg_nvme/lv_2... 2024-02-10T05:54:43.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:54:44.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:43 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:54:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:43 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:45.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:44 smithi146 ceph-mon[33795]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:54:45.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:44 smithi146 ceph-mon[33795]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:54:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:44 smithi146 ceph-mon[33795]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:44 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:44 smithi146 ceph-mon[33795]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:45.304 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[41707]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[41707]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[41707]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[41707]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[30020]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[30020]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:54:45.305 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[30020]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:54:45.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:45.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-mon[30020]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:54:45.306 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T05:54:44.910+0000 7fe416775700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T05:54:45.306 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:54:44 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T05:54:44.912+0000 7fe412568700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:54:46.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:54:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:54:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:54:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:46.176 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:54:46.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:54:46.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: from='osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:54:46.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:54:46.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:46.177 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:47.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[30020]: osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] boot 2024-02-10T05:54:47.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[30020]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:54:47.008 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:47.008 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[30020]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:47.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[41707]: osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] boot 2024-02-10T05:54:47.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[41707]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:54:47.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:47.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:46 smithi121 ceph-mon[41707]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:46 smithi146 ceph-mon[33795]: osd.1 [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] boot 2024-02-10T05:54:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:46 smithi146 ceph-mon[33795]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:54:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:54:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:46 smithi146 ceph-mon[33795]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0154771 s, 677 MB/s 2024-02-10T05:54:47.203 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-10T05:54:47.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-02-10T05:54:48.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:47 smithi146 ceph-mon[33795]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:54:48.273 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:47 smithi121 ceph-mon[41707]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:54:48.273 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:47 smithi121 ceph-mon[30020]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:54:48.648 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi121 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 8032M 2024-02-10T05:54:49.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi121 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 8032M 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.733 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi121 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 8032M 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:54:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 -- 172.21.15.121:0/3009492547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e01027f0 msgr2=0x7fd0e01090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3009492547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e01027f0 0x7fd0e01090a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b0a0 tx=0x7fd0d800b3b0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 -- 172.21.15.121:0/3009492547 shutdown_connections 2024-02-10T05:54:50.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3009492547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e01027f0 0x7fd0e01090a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3009492547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e0100c50 0x7fd0e0101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3009492547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0101e50 0x7fd0e01022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 -- 172.21.15.121:0/3009492547 >> 172.21.15.121:0/3009492547 conn(0x7fd0e00fc1f0 msgr2=0x7fd0e00fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 -- 172.21.15.121:0/3009492547 shutdown_connections 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.279+0000 7fd0e5506700 1 -- 172.21.15.121:0/3009492547 wait complete. 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 Processor -- start 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 -- start start 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 0x7fd0e019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e019df40 con 0x7fd0e0100c50 2024-02-10T05:54:50.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e019e080 con 0x7fd0e01027f0 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0e5506700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e019e210 con 0x7fd0e0101e50 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0de7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0de7fc700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:48648/0 (socket says 172.21.15.121:48648) 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0deffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48506/0 (socket says 172.21.15.121:48506) 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.280+0000 7fd0de7fc700 1 -- 172.21.15.121:0/3333372041 learned_addr learned my addr 172.21.15.121:0/3333372041 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0df7fe700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 0x7fd0e019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 -- 172.21.15.121:0/3333372041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 msgr2=0x7fd0e019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 0x7fd0e019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 -- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 msgr2=0x7fd0e0196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 -- 172.21.15.121:0/3333372041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d4009520 con 0x7fd0e0101e50 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0deffd700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:54:50.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0df7fe700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 0x7fd0e019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0de7fc700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400eb60 tx=0x7fd0d400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400cc90 con 0x7fd0e0101e50 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d8009d00 con 0x7fd0e0101e50 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd0d40044a0 con 0x7fd0e0101e50 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d4003e20 con 0x7fd0e0101e50 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.281+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e019e930 con 0x7fd0e0101e50 2024-02-10T05:54:50.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.282+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0107830 con 0x7fd0e0101e50 2024-02-10T05:54:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.282+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 14) v1 ==== 91351+0+0 (secure 0 0 0) 0x7fd0d4003f80 con 0x7fd0e0101e50 2024-02-10T05:54:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.283+0000 7fd0c7fff700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 0x7fd0c806fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:54:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.283+0000 7fd0deffd700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 0x7fd0c806fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:54:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.283+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7fd0d4014070 con 0x7fd0e0101e50 2024-02-10T05:54:50.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.283+0000 7fd0deffd700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 0x7fd0c806fe00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0101cb0 tx=0x7fd0d0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:54:50.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.288+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd0d4056f40 con 0x7fd0e0101e50 2024-02-10T05:54:50.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:54:50.491+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0e010bca0 con 0x7fd0c806d940 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:51.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:51 smithi121 ceph-mon[30020]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.681 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:51 smithi121 ceph-mon[30020]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:51.681 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:51 smithi121 ceph-mon[41707]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.681 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:51 smithi121 ceph-mon[41707]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:51 smithi146 ceph-mon[33795]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:51 smithi146 ceph-mon[33795]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/656515782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]': finished 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/656515782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]': finished 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:54:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:54:54.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:54.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/656515782' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]: dispatch 2024-02-10T05:54:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104"}]': finished 2024-02-10T05:54:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:54:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:54:55.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1281037085' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1281037085' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:55.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1281037085' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:56.025 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:55 smithi121 ceph-mon[30020]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:56.026 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:55 smithi121 ceph-mon[41707]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:56.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:55 smithi146 ceph-mon[33795]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:57.948 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:57 smithi121 ceph-mon[30020]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:57.949 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:57 smithi121 ceph-mon[41707]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:58.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:57 smithi146 ceph-mon[33795]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:58 smithi146 ceph-mon[33795]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:59.293 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:58 smithi121 ceph-mon[30020]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:54:59.294 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:58 smithi121 ceph-mon[41707]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:59 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:55:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:59 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:54:59 smithi146 ceph-mon[33795]: Deploying daemon osd.2 on smithi121 2024-02-10T05:55:00.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:55:00.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:00.182 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[30020]: Deploying daemon osd.2 on smithi121 2024-02-10T05:55:00.183 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:55:00.183 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:00.183 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:54:59 smithi121 ceph-mon[41707]: Deploying daemon osd.2 on smithi121 2024-02-10T05:55:01.167 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:00 smithi121 ceph-mon[30020]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:01.167 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:00 smithi121 ceph-mon[41707]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:01.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:00 smithi146 ceph-mon[33795]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:03 smithi121 ceph-mon[30020]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:03 smithi121 ceph-mon[41707]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:03 smithi146 ceph-mon[33795]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[30020]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[41707]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:05 smithi146 ceph-mon[33795]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:07 smithi121 ceph-mon[30020]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:07 smithi121 ceph-mon[41707]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:07 smithi146 ceph-mon[33795]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:08.893 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.892+0000 7fd0c7fff700 1 -- 172.21.15.121:0/3333372041 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd0e010bca0 con 0x7fd0c806d940 2024-02-10T05:55:08.893 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 2 on host 'smithi121' 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 msgr2=0x7fd0c806fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 0x7fd0c806fe00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0101cb0 tx=0x7fd0d0009190 comp rx=0 tx=0).stop 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 msgr2=0x7fd0e0196940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0d400eb60 tx=0x7fd0d400ee70 comp rx=0 tx=0).stop 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 shutdown_connections 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0e01027f0 0x7fd0e019d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:08.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e0100c50 0x7fd0e0196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd0c806d940 0x7fd0c806fe00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.894+0000 7fd0e5506700 1 --2- 172.21.15.121:0/3333372041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e0101e50 0x7fd0e0196940 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.895+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 >> 172.21.15.121:0/3333372041 conn(0x7fd0e00fc1f0 msgr2=0x7fd0e00fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:55:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.895+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 shutdown_connections 2024-02-10T05:55:08.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:08.895+0000 7fd0e5506700 1 -- 172.21.15.121:0/3333372041 wait complete. 2024-02-10T05:55:09.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:55:09.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:08 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.271 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:09.272 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.273 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:08 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:09.603 DEBUG:teuthology.orchestra.run.smithi121:osd.2> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T05:55:09.605 INFO:tasks.cephadm:Deploying osd.3 on smithi121 with /dev/vg_nvme/lv_1... 2024-02-10T05:55:09.605 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:55:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:09 smithi146 ceph-mon[33795]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:55:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:09 smithi146 ceph-mon[33795]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:55:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:09 smithi146 ceph-mon[33795]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:10.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:09 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:10.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[30020]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[30020]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[30020]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[41707]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[41707]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[41707]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:10.330 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2[58537]: 2024-02-10T05:55:09.914+0000 7f6a6b62f700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T05:55:10.330 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2[58537]: 2024-02-10T05:55:09.919+0000 7f6a67422700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:55:10.657 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:10 smithi146 ceph-mon[33795]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:10 smithi146 ceph-mon[33795]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:55:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:10 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:10 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:10 smithi146 ceph-mon[33795]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:11.256 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[30020]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[30020]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[30020]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[41707]: from='osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[41707]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.257 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.258 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:10 smithi121 ceph-mon[41707]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:55:11.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:55:11.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:55:11.970 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] boot 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] boot 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:55:11.971 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:11 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:12.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:55:12.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:55:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: osd.2 [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] boot 2024-02-10T05:55:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:55:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:11 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:13.092 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:55:13.093 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:55:13.093 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-02-10T05:55:13.093 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-02-10T05:55:13.093 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117665 s, 891 MB/s 2024-02-10T05:55:13.093 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-02-10T05:55:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:13 smithi121 ceph-mon[30020]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:55:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:13 smithi121 ceph-mon[41707]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:55:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:13 smithi146 ceph-mon[33795]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi121 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 5354M 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi121 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 5354M 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:14 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.228 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: pgmap v79: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi121 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 5354M 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:14.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:14 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:14.467 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:14.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62428]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:14.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62428]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:14.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62428]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:14.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62428]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:14.846 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62485]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:14.846 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62485]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:14.846 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62485]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:14.846 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62485]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:15.185 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[41707]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[41707]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[30020]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:15.186 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 ceph-mon[30020]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:15.186 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62525]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:15.187 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62525]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:15.187 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62525]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:15.187 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:14 smithi121 sudo[62525]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:55:15.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:15 smithi146 ceph-mon[33795]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:55:15.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:15.435 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:15 smithi146 ceph-mon[33795]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:15.447 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62539]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:15.447 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:15.447 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62539]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:15.448 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62528]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:15.448 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62528]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:15.448 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62528]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:15.448 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62528]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:15.700 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62539]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:15.700 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62561]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:15.700 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62561]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:15.700 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62561]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:15.701 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62561]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:15.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62585]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:15.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:15.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62585]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:15.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:15 smithi121 sudo[62585]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:16.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 -- 172.21.15.121:0/3042962157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0101e80 msgr2=0x7f37b01022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/3042962157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0101e80 0x7f37b01022e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009990 tx=0x7f37a0009ca0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 -- 172.21.15.121:0/3042962157 shutdown_connections 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/3042962157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0102820 0x7f37b01090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/3042962157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0100c80 0x7f37b01010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/3042962157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0101e80 0x7f37b01022e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 -- 172.21.15.121:0/3042962157 >> 172.21.15.121:0/3042962157 conn(0x7f37b00fc220 msgr2=0x7f37b00fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 -- 172.21.15.121:0/3042962157 shutdown_connections 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.092+0000 7f37b7fe3700 1 -- 172.21.15.121:0/3042962157 wait complete. 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 Processor -- start 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 -- start start 2024-02-10T05:55:16.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0101e80 0x7f37b019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0102820 0x7f37b019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019ed40 con 0x7f37b0102820 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019eeb0 con 0x7f37b0101e80 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b7fe3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b019f070 con 0x7f37b0100c80 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b5d7f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b5d7f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:46410/0 (socket says 172.21.15.121:46410) 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b5d7f700 1 -- 172.21.15.121:0/1622681049 learned_addr learned my addr 172.21.15.121:0/1622681049 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b6580700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0102820 0x7f37b019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.093+0000 7f37b557e700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0101e80 0x7f37b019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:16.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 -- 172.21.15.121:0/1622681049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0101e80 msgr2=0x7f37b019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0101e80 0x7f37b019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 -- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0102820 msgr2=0x7f37b019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0102820 0x7f37b019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 -- 172.21.15.121:0/1622681049 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00095f0 con 0x7f37b0100c80 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b5d7f700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00d980 tx=0x7f37ac00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac004020 con 0x7f37b0100c80 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b019f350 con 0x7f37b0100c80 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f37ac009730 con 0x7f37b0100c80 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac003a20 con 0x7f37b0100c80 2024-02-10T05:55:16.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.094+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b019f8a0 con 0x7f37b0100c80 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.095+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b0107860 con 0x7f37b0100c80 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.096+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 14) v1 ==== 91351+0+0 (secure 0 0 0) 0x7f37ac004180 con 0x7f37b0100c80 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.096+0000 7f37a6ffd700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 0x7f379c06fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.096+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3017+0+0 (secure 0 0 0) 0x7f37ac08c190 con 0x7f37b0100c80 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.096+0000 7f37b557e700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 0x7f379c06fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:16.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.096+0000 7f37b557e700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 0x7f379c06fe00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37a000b680 tx=0x7f37a000b610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:55:16.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.101+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37ac056810 con 0x7f37b0100c80 2024-02-10T05:55:16.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:16.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62615]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:16.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:16.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62615]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:16.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.227+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f37ac056be0 con 0x7f37b0100c80 2024-02-10T05:55:16.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:16.311+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f37b0062160 con 0x7f379c06d940 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 ceph-mon[33795]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36615]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:16.480 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36615]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 ceph-mon[30020]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:55:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:16 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:16.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:16 smithi121 ceph-mon[41707]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:55:16.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:16 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36618]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36618]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36618]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:16 smithi146 sudo[36618]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:17.247 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62741]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:55:17.248 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:17.248 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:16 smithi121 sudo[62741]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:17.248 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 sudo[62741]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:17.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 sudo[62782]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 sudo[62782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 sudo[62782]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 sudo[62782]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:18.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:18 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:18.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:18 smithi121 ceph-mon[30020]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:18.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:18 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:18.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:18 smithi121 ceph-mon[41707]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:18.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:18 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:55:18.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:18 smithi146 ceph-mon[33795]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1879530462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]': finished 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:55:19.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1879530462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]': finished 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:55:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:19.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:19.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1879530462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]: dispatch 2024-02-10T05:55:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b613b780-0fb2-4edf-b40f-2bc3c5782e98"}]': finished 2024-02-10T05:55:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:55:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:20.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4086320759' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4086320759' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4086320759' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:21.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:21 smithi121 ceph-mon[30020]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:21 smithi121 ceph-mon[41707]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:21 smithi146 ceph-mon[33795]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:23 smithi121 ceph-mon[30020]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:55:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:23 smithi121 ceph-mon[41707]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:55:24.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:23 smithi146 ceph-mon[33795]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:55:26.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[30020]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:55:26.007 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:55:26.008 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:26.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[41707]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:55:26.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:55:26.008 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:25 smithi146 ceph-mon[33795]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:55:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:55:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:26.805 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:26 smithi121 ceph-mon[30020]: Deploying daemon osd.3 on smithi121 2024-02-10T05:55:26.805 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:26 smithi121 ceph-mon[41707]: Deploying daemon osd.3 on smithi121 2024-02-10T05:55:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:26 smithi146 ceph-mon[33795]: Deploying daemon osd.3 on smithi121 2024-02-10T05:55:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:27 smithi121 ceph-mon[30020]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:27 smithi121 ceph-mon[41707]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:27 smithi146 ceph-mon[33795]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:29.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:29 smithi121 ceph-mon[30020]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:29.098 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:29 smithi121 ceph-mon[41707]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:29.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:29 smithi146 ceph-mon[33795]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:55:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[30020]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:31.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:31 smithi121 ceph-mon[41707]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-02-10T05:55:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:31 smithi146 ceph-mon[33795]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:33.998 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[30020]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:33.999 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:33 smithi121 ceph-mon[41707]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:34.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:33 smithi146 ceph-mon[33795]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:55:34.414 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 3 on host 'smithi121' 2024-02-10T05:55:34.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.413+0000 7f37a6ffd700 1 -- 172.21.15.121:0/1622681049 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f37b0062160 con 0x7f379c06d940 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 msgr2=0x7f379c06fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 0x7f379c06fe00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37a000b680 tx=0x7f37a000b610 comp rx=0 tx=0).stop 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 msgr2=0x7f37b006af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00d980 tx=0x7f37ac00dc90 comp rx=0 tx=0).stop 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 shutdown_connections 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37b0102820 0x7f37b019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:34.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f379c06d940 0x7f379c06fe00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:34.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37b0100c80 0x7f37b006af40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:34.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 --2- 172.21.15.121:0/1622681049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37b0101e80 0x7f37b019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:34.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 >> 172.21.15.121:0/1622681049 conn(0x7f37b00fc220 msgr2=0x7f37b00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:55:34.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 shutdown_connections 2024-02-10T05:55:34.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:55:34.415+0000 7f37b7fe3700 1 -- 172.21.15.121:0/1622681049 wait complete. 2024-02-10T05:55:35.025 DEBUG:teuthology.orchestra.run.smithi121:osd.3> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T05:55:35.027 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_4... 2024-02-10T05:55:35.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T05:55:35.276 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[41707]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-mon[30020]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:35.626 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:35.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:55:35.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.627 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:55:35.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:35.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-02-10T05:55:35.628 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:35 smithi146 ceph-mon[33795]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:35.917 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T05:55:35.616+0000 7f79c9c5e700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T05:55:35.917 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 05:55:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T05:55:35.621+0000 7f79c5a51700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:55:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[30020]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[30020]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[41707]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[41707]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:36 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:36 smithi146 ceph-mon[33795]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-02-10T05:55:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:36 smithi146 ceph-mon[33795]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T05:55:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:36 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:36 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.195 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T05:55:37.195 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T05:55:37.195 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-02-10T05:55:37.196 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-02-10T05:55:37.196 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112954 s, 928 MB/s 2024-02-10T05:55:37.196 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-02-10T05:55:37.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:55:37.796 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:55:37.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] boot 2024-02-10T05:55:37.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:55:37.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.797 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:37 smithi146 ceph-mon[33795]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:37.823 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] boot 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[30020]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:55:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:55:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: osd.3 [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] boot 2024-02-10T05:55:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:55:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:37 smithi121 ceph-mon[41707]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:55:38.061 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:38.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T05:55:38.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:38.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:38.903 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:38.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T05:55:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T05:55:39.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:39.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 -- 172.21.15.146:0/1430165140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac101ec0 msgr2=0x7f8eac102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 --2- 172.21.15.146:0/1430165140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac101ec0 0x7f8eac102320 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0099c0 tx=0x7f8e9c009cd0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 -- 172.21.15.146:0/1430165140 shutdown_connections 2024-02-10T05:55:39.688 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 --2- 172.21.15.146:0/1430165140 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac102860 0x7f8eac109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 --2- 172.21.15.146:0/1430165140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 0x7f8eac1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 --2- 172.21.15.146:0/1430165140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac101ec0 0x7f8eac102320 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 -- 172.21.15.146:0/1430165140 >> 172.21.15.146:0/1430165140 conn(0x7f8eac0fc280 msgr2=0x7f8eac0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 -- 172.21.15.146:0/1430165140 shutdown_connections 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.685+0000 7f8eb3467700 1 -- 172.21.15.146:0/1430165140 wait complete. 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 Processor -- start 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 -- start start 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 0x7f8eac197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac101ec0 0x7f8eac1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac19ee10 con 0x7f8eac100cc0 2024-02-10T05:55:39.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac19ef50 con 0x7f8eac102860 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb3467700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac19f0e0 con 0x7f8eac101ec0 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb1a04700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb1a04700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:55954/0 (socket says 172.21.15.146:55954) 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb1a04700 1 -- 172.21.15.146:0/4011870698 learned_addr learned my addr 172.21.15.146:0/4011870698 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb1203700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 0x7f8eac197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.686+0000 7f8eb0a02700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac101ec0 0x7f8eac1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 -- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac101ec0 msgr2=0x7f8eac1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac101ec0 0x7f8eac1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 -- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 msgr2=0x7f8eac197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 0x7f8eac197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 -- 172.21.15.146:0/4011870698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9c0095f0 con 0x7f8eac102860 2024-02-10T05:55:39.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb1a04700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4007ee0 tx=0x7f8ea400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:55:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4017070 con 0x7f8eac102860 2024-02-10T05:55:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac19f3c0 con 0x7f8eac102860 2024-02-10T05:55:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8ea400f040 con 0x7f8eac102860 2024-02-10T05:55:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4005530 con 0x7f8eac102860 2024-02-10T05:55:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.687+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac19f8c0 con 0x7f8eac102860 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.688+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac108150 con 0x7f8eac102860 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.689+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f8ea400db20 con 0x7f8eac102860 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.689+0000 7f8ea27fc700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 0x7f8e9806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.689+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3449+0+0 (secure 0 0 0) 0x7f8ea408be70 con 0x7f8eac102860 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.689+0000 7f8eb1203700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 0x7f8e9806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:55:39.692 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.690+0000 7f8eb1203700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 0x7f8e9806fe90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8eac101d20 tx=0x7f8ea800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:55:39.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.694+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8ea4056260 con 0x7f8eac102860 2024-02-10T05:55:39.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:39.895+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eac10cf20 con 0x7f8e9806d9d0 2024-02-10T05:55:39.895 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:39 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi121 2024-02-10T05:55:39.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:39 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 4016M 2024-02-10T05:55:39.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:39 smithi146 ceph-mon[33795]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi121 2024-02-10T05:55:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 4016M 2024-02-10T05:55:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[41707]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:40.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi121 2024-02-10T05:55:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 4016M 2024-02-10T05:55:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:39 smithi121 ceph-mon[30020]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:40.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:40.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:40.741 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:40 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:55:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:55:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:40 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:41.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:41 smithi146 ceph-mon[33795]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:41.738 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:41 smithi146 ceph-mon[33795]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:42.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:41 smithi121 ceph-mon[30020]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:42.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:41 smithi121 ceph-mon[30020]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:42.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:41 smithi121 ceph-mon[41707]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:42.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:41 smithi121 ceph-mon[41707]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:42.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/4253477326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:42.913 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:42.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]': finished 2024-02-10T05:55:42.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:42 smithi146 ceph-mon[33795]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:55:42.914 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:42 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/4253477326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]': finished 2024-02-10T05:55:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[30020]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:55:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:43.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/4253477326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:43.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]: dispatch 2024-02-10T05:55:43.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3bf95a4e-2fe0-4799-8e53-e730e7f49b66"}]': finished 2024-02-10T05:55:43.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[41707]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:55:43.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:42 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:43.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:43 smithi146 ceph-mon[33795]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:43.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/616778109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:43 smithi121 ceph-mon[30020]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/616778109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:43 smithi121 ceph-mon[41707]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/616778109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:55:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:45 smithi121 ceph-mon[30020]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:45 smithi121 ceph-mon[41707]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:46.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:45 smithi146 ceph-mon[33795]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:47 smithi121 ceph-mon[41707]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:47 smithi121 ceph-mon[30020]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:48.102 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:47 smithi146 ceph-mon[33795]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:49.159 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[41707]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:49.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:55:49.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[30020]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:55:49.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:49 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:49.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:49 smithi146 ceph-mon[33795]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:49.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:55:49.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:49 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:50.320 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:50 smithi146 ceph-mon[33795]: Deploying daemon osd.4 on smithi146 2024-02-10T05:55:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:50 smithi121 ceph-mon[41707]: Deploying daemon osd.4 on smithi146 2024-02-10T05:55:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:50 smithi121 ceph-mon[30020]: Deploying daemon osd.4 on smithi146 2024-02-10T05:55:51.292 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:51 smithi146 ceph-mon[33795]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:51.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:51 smithi121 ceph-mon[30020]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:51 smithi121 ceph-mon[41707]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:53.900 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:53 smithi146 ceph-mon[33795]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:53 smithi121 ceph-mon[30020]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:53 smithi121 ceph-mon[41707]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[30020]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:55.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:55 smithi121 ceph-mon[41707]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:55.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:55 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:55 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:55 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:55 smithi146 ceph-mon[33795]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:57.611 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:57 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:57.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:57 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.309 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.308+0000 7f8ea27fc700 1 -- 172.21.15.146:0/4011870698 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8eac10cf20 con 0x7f8e9806d9d0 2024-02-10T05:55:58.309 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 msgr2=0x7f8e9806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 0x7f8e9806fe90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8eac101d20 tx=0x7f8ea800b410 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 msgr2=0x7f8eac19e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4007ee0 tx=0x7f8ea400d3b0 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 shutdown_connections 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8eac102860 0x7f8eac19e840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8eac100cc0 0x7f8eac197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f8e9806d9d0 0x7f8e9806fe90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 --2- 172.21.15.146:0/4011870698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8eac101ec0 0x7f8eac1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 >> 172.21.15.146:0/4011870698 conn(0x7f8eac0fc280 msgr2=0x7f8eac105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 shutdown_connections 2024-02-10T05:55:58.311 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:55:58.310+0000 7f8eb3467700 1 -- 172.21.15.146:0/4011870698 wait complete. 2024-02-10T05:55:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[30020]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[30020]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[41707]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[41707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:58.896 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:58 smithi146 ceph-mon[33795]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:58 smithi146 ceph-mon[33795]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:55:58.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:55:58.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:58.926 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T05:55:58.928 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_3... 2024-02-10T05:55:58.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T05:55:59.692 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.814 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-mon[33795]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:59.815 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T05:55:59.491+0000 7f7a2a550700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T05:55:59.815 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 05:55:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T05:55:59.497+0000 7f7a26343700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[30020]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:55:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:55:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:55:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: from='osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:55:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:55:59 smithi121 ceph-mon[41707]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-10T05:56:00.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:00 smithi146 ceph-mon[33795]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:00.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:00 smithi146 ceph-mon[33795]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T05:56:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[30020]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[30020]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T05:56:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[41707]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[41707]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T05:56:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:01.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] boot 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:01 smithi146 ceph-mon[33795]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:56:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:56:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:56:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] boot 2024-02-10T05:56:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:56:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[30020]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: osd.4 [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] boot 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:01 smithi121 ceph-mon[41707]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:56:02.319 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T05:56:02.319 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T05:56:02.320 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-02-10T05:56:02.320 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-02-10T05:56:02.320 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123082 s, 852 MB/s 2024-02-10T05:56:02.320 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-02-10T05:56:03.053 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-02-10T05:56:03.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:03 smithi146 ceph-mon[33795]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:56:03.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:03 smithi146 ceph-mon[33795]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:03.622 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:03 smithi146 ceph-mon[33795]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:56:03.721 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:56:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[30020]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:56:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[30020]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[30020]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:56:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[41707]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:56:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[41707]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:03.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:03 smithi121 ceph-mon[41707]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:56:04.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi146 2024-02-10T05:56:04.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:04.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:04.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:04.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 17088M 2024-02-10T05:56:04.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:04.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:04.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:04.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:04.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:04 smithi146 ceph-mon[33795]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi146 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 17088M 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[30020]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi146 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 17088M 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:05.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:04 smithi121 ceph-mon[41707]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 -- 172.21.15.146:0/3825084372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898101e50 msgr2=0x7fb8981022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 --2- 172.21.15.146:0/3825084372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898101e50 0x7fb8981022b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb88c0099c0 tx=0x7fb88c009cd0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 -- 172.21.15.146:0/3825084372 shutdown_connections 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 --2- 172.21.15.146:0/3825084372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb8981090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 --2- 172.21.15.146:0/3825084372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898100c50 0x7fb898101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 --2- 172.21.15.146:0/3825084372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898101e50 0x7fb8981022b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 -- 172.21.15.146:0/3825084372 >> 172.21.15.146:0/3825084372 conn(0x7fb8980fc1f0 msgr2=0x7fb8980fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 -- 172.21.15.146:0/3825084372 shutdown_connections 2024-02-10T05:56:05.412 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.411+0000 7fb89c868700 1 -- 172.21.15.146:0/3825084372 wait complete. 2024-02-10T05:56:05.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 Processor -- start 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 -- start start 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 0x7fb89819c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb89819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89819ef00 con 0x7fb8981027f0 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89819f070 con 0x7fb898100c50 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89c868700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89819f230 con 0x7fb898101e50 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:38096/0 (socket says 172.21.15.146:38096) 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 -- 172.21.15.146:0/2756088589 learned_addr learned my addr 172.21.15.146:0/2756088589 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb895d9b700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 0x7fb89819c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:05.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb896d9d700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb89819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 -- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 msgr2=0x7fb89819c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 0x7fb89819c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 -- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 msgr2=0x7fb89819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb89819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb89659c700 1 -- 172.21.15.146:0/2756088589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0095f0 con 0x7fb898100c50 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb895d9b700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 0x7fb89819c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.412+0000 7fb896d9d700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb89819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb89659c700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b500 tx=0x7fb88800b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888011830 con 0x7fb898100c50 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb888011e10 con 0x7fb898100c50 2024-02-10T05:56:05.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89819f510 con 0x7fb898100c50 2024-02-10T05:56:05.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888010d50 con 0x7fb898100c50 2024-02-10T05:56:05.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.413+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89819fa90 con 0x7fb898100c50 2024-02-10T05:56:05.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.414+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb898107830 con 0x7fb898100c50 2024-02-10T05:56:05.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.414+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7fb888011990 con 0x7fb898100c50 2024-02-10T05:56:05.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.415+0000 7fb8877fe700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 0x7fb88006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:05.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.415+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 3991+0+0 (secure 0 0 0) 0x7fb88808c560 con 0x7fb898100c50 2024-02-10T05:56:05.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.415+0000 7fb895d9b700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 0x7fb88006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:05.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.415+0000 7fb895d9b700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 0x7fb88006fe90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb88c006010 tx=0x7fb88c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:05.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.419+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb888056730 con 0x7fb898100c50 2024-02-10T05:56:05.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:05.621+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb898062160 con 0x7fb88006d9d0 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[30020]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[41707]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:06.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:06.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:05 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:05 smithi146 ceph-mon[33795]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:06.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:06.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:06.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:05 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:06.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:06 smithi146 ceph-mon[33795]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:06 smithi121 ceph-mon[30020]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:06 smithi121 ceph-mon[41707]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:07 smithi146 ceph-mon[33795]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-02-10T05:56:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:07 smithi121 ceph-mon[30020]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-02-10T05:56:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:07 smithi121 ceph-mon[41707]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-02-10T05:56:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/2226279662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]': finished 2024-02-10T05:56:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:56:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/198010137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/2226279662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]': finished 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:56:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/198010137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/2226279662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]: dispatch 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55de8feb-ec6c-4512-b30a-2275e97e8509"}]': finished 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:56:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/198010137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:11 smithi121 ceph-mon[30020]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:56:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:11 smithi121 ceph-mon[41707]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:56:12.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:11 smithi146 ceph-mon[33795]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:56:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:13 smithi146 ceph-mon[33795]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-10T05:56:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:13 smithi121 ceph-mon[30020]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-10T05:56:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:13 smithi121 ceph-mon[41707]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[30020]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[30020]: Deploying daemon osd.5 on smithi146 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[41707]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:15 smithi121 ceph-mon[41707]: Deploying daemon osd.5 on smithi146 2024-02-10T05:56:16.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:15 smithi146 ceph-mon[33795]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:16.111 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:56:16.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:16.112 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:15 smithi146 ceph-mon[33795]: Deploying daemon osd.5 on smithi146 2024-02-10T05:56:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:17 smithi121 ceph-mon[30020]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:17 smithi121 ceph-mon[41707]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:18.130 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:17 smithi146 ceph-mon[33795]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:19.154 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:19 smithi121 ceph-mon[30020]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:19.154 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:19 smithi121 ceph-mon[41707]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:19 smithi146 ceph-mon[33795]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:21 smithi146 ceph-mon[33795]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[30020]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:21.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:21.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:21.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:21 smithi121 ceph-mon[41707]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[30020]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:23 smithi121 ceph-mon[41707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:23 smithi146 ceph-mon[33795]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:56:24.235 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.234+0000 7fb8877fe700 1 -- 172.21.15.146:0/2756088589 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb898062160 con 0x7fb88006d9d0 2024-02-10T05:56:24.235 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 msgr2=0x7fb88006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 0x7fb88006fe90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb88c006010 tx=0x7fb88c000bc0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 msgr2=0x7fb89806a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb88800b500 tx=0x7fb88800b8c0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 shutdown_connections 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8981027f0 0x7fb89819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb898100c50 0x7fb89806a550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb88006d9d0 0x7fb88006fe90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 --2- 172.21.15.146:0/2756088589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb898101e50 0x7fb89819c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:24.237 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 >> 172.21.15.146:0/2756088589 conn(0x7fb8980fc1f0 msgr2=0x7fb8980fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:56:24.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.236+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 shutdown_connections 2024-02-10T05:56:24.238 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:24.237+0000 7fb89c868700 1 -- 172.21.15.146:0/2756088589 wait complete. 2024-02-10T05:56:24.919 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T05:56:24.922 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-02-10T05:56:24.922 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T05:56:25.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-mon[33795]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:25.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[30020]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:25.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:25.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:25 smithi121 ceph-mon[41707]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:56:26.059 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T05:56:25.770+0000 7f4829c59700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T05:56:26.059 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 05:56:25 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T05:56:25.774+0000 7f4821a44700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:56:26.148 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:56:26.635 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:26 smithi146 ceph-mon[33795]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:26.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:26 smithi146 ceph-mon[33795]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T05:56:26.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:26 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:26 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.636 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:26 smithi146 ceph-mon[33795]: from='osd.5 ' entity='osd.5' 2024-02-10T05:56:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[30020]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[30020]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T05:56:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[30020]: from='osd.5 ' entity='osd.5' 2024-02-10T05:56:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[41707]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[41707]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T05:56:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:26 smithi121 ceph-mon[41707]: from='osd.5 ' entity='osd.5' 2024-02-10T05:56:27.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] boot 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:27 smithi146 ceph-mon[33795]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] boot 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[30020]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: osd.5 [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] boot 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: pgmap v136: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:27 smithi121 ceph-mon[41707]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:56:29.259 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T05:56:29.259 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T05:56:29.259 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-02-10T05:56:29.259 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-02-10T05:56:29.260 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012241 s, 857 MB/s 2024-02-10T05:56:29.260 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi146 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 8544M 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:29.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi146 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.572 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 8544M 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:29 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:56:29.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi146 2024-02-10T05:56:29.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 8544M 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: pgmap v139: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:29.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:29 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:29.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-02-10T05:56:30.200 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:56:30.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:30 smithi121 ceph-mon[30020]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:56:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:30 smithi121 ceph-mon[41707]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:56:30.631 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:30 smithi146 ceph-mon[33795]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:56:31.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:31 smithi146 ceph-mon[33795]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T05:56:31.555 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:31 smithi146 ceph-mon[33795]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- 172.21.15.146:0/706545587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 msgr2=0x7fd800102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 --2- 172.21.15.146:0/706545587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800102160 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7f00099c0 tx=0x7fd7f0009cd0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- 172.21.15.146:0/706545587 shutdown_connections 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 --2- 172.21.15.146:0/706545587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8001038e0 0x7fd80010a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 --2- 172.21.15.146:0/706545587 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800102160 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 --2- 172.21.15.146:0/706545587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 0x7fd8001033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- 172.21.15.146:0/706545587 >> 172.21.15.146:0/706545587 conn(0x7fd8000fd300 msgr2=0x7fd8000ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- 172.21.15.146:0/706545587 shutdown_connections 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- 172.21.15.146:0/706545587 wait complete. 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 Processor -- start 2024-02-10T05:56:31.777 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.776+0000 7fd806459700 1 -- start start 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 0x7fd800074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8001038e0 0x7fd800074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800074dd0 con 0x7fd800102f40 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800074f40 con 0x7fd800101d40 2024-02-10T05:56:31.779 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8001a13b0 con 0x7fd8001038e0 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:51324/0 (socket says 172.21.15.146:51324) 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 -- 172.21.15.146:0/3784679658 learned_addr learned my addr 172.21.15.146:0/3784679658 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 -- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8001038e0 msgr2=0x7fd800074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8001038e0 0x7fd800074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 -- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 msgr2=0x7fd800074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7ff7fe700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 0x7fd800074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 0x7fd800074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 -- 172.21.15.146:0/3784679658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00095f0 con 0x7fd800101d40 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fffff700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0000c00 tx=0x7fd7f000f7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f001d070 con 0x7fd800101d40 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd7f000bd30 con 0x7fd800101d40 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8001a1630 con 0x7fd800101d40 2024-02-10T05:56:31.780 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0022d60 con 0x7fd800101d40 2024-02-10T05:56:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.777+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001a1ba0 con 0x7fd800101d40 2024-02-10T05:56:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.778+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd800108920 con 0x7fd800101d40 2024-02-10T05:56:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.779+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7fd7f000fcb0 con 0x7fd800101d40 2024-02-10T05:56:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.779+0000 7fd7fd7fa700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 0x7fd7e806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:31.781 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.779+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4423+0+0 (secure 0 0 0) 0x7fd7f008df70 con 0x7fd800101d40 2024-02-10T05:56:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.780+0000 7fd7ff7fe700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 0x7fd7e806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:31.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.780+0000 7fd7ff7fe700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 0x7fd7e806fe90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4006fd0 tx=0x7fd7f4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:31.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.784+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd7f00c1d20 con 0x7fd800101d40 2024-02-10T05:56:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:31 smithi121 ceph-mon[30020]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T05:56:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:31 smithi121 ceph-mon[30020]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:31 smithi121 ceph-mon[41707]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T05:56:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:31 smithi121 ceph-mon[41707]: pgmap v142: 1 pgs: 1 unknown; 577 KiB data, 382 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:31.987 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:31.986+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd800062160 con 0x7fd7e806d9d0 2024-02-10T05:56:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:32 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:32 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:32 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:32 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:33.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:33 smithi146 ceph-mon[33795]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:33 smithi146 ceph-mon[33795]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-02-10T05:56:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:33 smithi121 ceph-mon[30020]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:33 smithi121 ceph-mon[30020]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-02-10T05:56:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:33 smithi121 ceph-mon[41707]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:33 smithi121 ceph-mon[41707]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-02-10T05:56:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/2105470201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]': finished 2024-02-10T05:56:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[41707]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/2105470201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]': finished 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[30020]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:56:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:34 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:35.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/2105470201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]: dispatch 2024-02-10T05:56:35.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c45f670b-7a69-4f2a-98ae-ce1fba25a540"}]': finished 2024-02-10T05:56:35.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:34 smithi146 ceph-mon[33795]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:56:35.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:34 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:35 smithi121 ceph-mon[30020]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-10T05:56:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/1766088048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:35 smithi121 ceph-mon[41707]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-10T05:56:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/1766088048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:35 smithi146 ceph-mon[33795]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-10T05:56:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/1766088048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:56:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:37 smithi146 ceph-mon[33795]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:56:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:37 smithi121 ceph-mon[30020]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:56:38.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:37 smithi121 ceph-mon[41707]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:56:39.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:39 smithi146 ceph-mon[33795]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:56:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:39 smithi121 ceph-mon[30020]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:56:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:39 smithi121 ceph-mon[41707]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:56:42.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[30020]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:42.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:56:42.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:42.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[41707]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:42.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:56:42.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:42.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:41 smithi146 ceph-mon[33795]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:56:42.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:56:42.109 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:42.855 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:42 smithi146 ceph-mon[33795]: Deploying daemon osd.6 on smithi146 2024-02-10T05:56:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:42 smithi121 ceph-mon[30020]: Deploying daemon osd.6 on smithi146 2024-02-10T05:56:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:42 smithi121 ceph-mon[41707]: Deploying daemon osd.6 on smithi146 2024-02-10T05:56:43.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:43 smithi146 ceph-mon[33795]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:43 smithi121 ceph-mon[30020]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:43 smithi121 ceph-mon[41707]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:45.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:45 smithi146 ceph-mon[33795]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:45 smithi121 ceph-mon[30020]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:45 smithi121 ceph-mon[41707]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[30020]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[41707]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:48.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:47 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:48.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:47 smithi146 ceph-mon[33795]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:47 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:47 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:47 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:49.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:49 smithi121 ceph-mon[30020]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:49 smithi121 ceph-mon[41707]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:49.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:49 smithi146 ceph-mon[33795]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:50.600 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.599+0000 7fd7fd7fa700 1 -- 172.21.15.146:0/3784679658 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd800062160 con 0x7fd7e806d9d0 2024-02-10T05:56:50.601 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-02-10T05:56:50.602 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 msgr2=0x7fd7e806fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 0x7fd7e806fe90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4006fd0 tx=0x7fd7f4008040 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 msgr2=0x7fd800075ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0000c00 tx=0x7fd7f000f7c0 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 shutdown_connections 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd7e806d9d0 0x7fd7e806fe90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8001038e0 0x7fd800074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd800101d40 0x7fd800075ac0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 --2- 172.21.15.146:0/3784679658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800102f40 0x7fd800074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 >> 172.21.15.146:0/3784679658 conn(0x7fd8000fd300 msgr2=0x7fd8000fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 shutdown_connections 2024-02-10T05:56:50.603 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:50.602+0000 7fd806459700 1 -- 172.21.15.146:0/3784679658 wait complete. 2024-02-10T05:56:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:51.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:51.226 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T05:56:51.228 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_1... 2024-02-10T05:56:51.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T05:56:52.004 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:56:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:56:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:56:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[30020]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:51 smithi121 ceph-mon[41707]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.113 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:52.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:56:52.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:56:52.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:52.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: from='osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.114 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-mon[33795]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:56:52.114 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T05:56:51.753+0000 7ffb3173c700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T05:56:52.114 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 05:56:51 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T05:56:51.759+0000 7ffb2d52f700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:56:53.035 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:52 smithi146 ceph-mon[33795]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:53.035 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:52 smithi146 ceph-mon[33795]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T05:56:53.035 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:53.036 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[30020]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[30020]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T05:56:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[41707]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:56:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[41707]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T05:56:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] boot 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.007 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:53 smithi146 ceph-mon[33795]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] boot 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[41707]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: osd.6 [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] boot 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:56:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:53 smithi121 ceph-mon[30020]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121566 s, 863 MB/s 2024-02-10T05:56:54.577 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-02-10T05:56:55.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:55 smithi146 ceph-mon[33795]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:56:55.240 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:55 smithi146 ceph-mon[33795]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:55.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-02-10T05:56:55.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:55 smithi121 ceph-mon[30020]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:56:55.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:55 smithi121 ceph-mon[30020]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:55 smithi121 ceph-mon[41707]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:56:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:55 smithi121 ceph-mon[41707]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:55.514 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:56 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:56 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:56:57.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:57 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi146 2024-02-10T05:56:57.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:57 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 5696M 2024-02-10T05:56:57.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:57 smithi146 ceph-mon[33795]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T05:56:57.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:57 smithi146 ceph-mon[33795]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi146 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 5696M 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[30020]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[30020]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi146 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 5696M 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[41707]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T05:56:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:57 smithi121 ceph-mon[41707]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 -- 172.21.15.146:0/435058622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50941027d0 msgr2=0x7f5094104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 --2- 172.21.15.146:0/435058622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50941027d0 0x7f5094104bc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5084009990 tx=0x7f5084009ca0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 -- 172.21.15.146:0/435058622 shutdown_connections 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 --2- 172.21.15.146:0/435058622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094107a30 0x7f5094109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 --2- 172.21.15.146:0/435058622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50941027d0 0x7f5094104bc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 --2- 172.21.15.146:0/435058622 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5094105100 0x7f50941074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 -- 172.21.15.146:0/435058622 >> 172.21.15.146:0/435058622 conn(0x7f50940fc220 msgr2=0x7f50940fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 -- 172.21.15.146:0/435058622 shutdown_connections 2024-02-10T05:56:57.526 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 -- 172.21.15.146:0/435058622 wait complete. 2024-02-10T05:56:57.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.525+0000 7f5099cc2700 1 Processor -- start 2024-02-10T05:56:57.527 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 -- start start 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50941027d0 0x7f5094197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094105100 0x7f50941978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509419eed0 con 0x7f50941027d0 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509419f010 con 0x7f5094107a30 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5099cc2700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509419f1a0 con 0x7f5094105100 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:50184/0 (socket says 172.21.15.146:50184) 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 -- 172.21.15.146:0/2499191279 learned_addr learned my addr 172.21.15.146:0/2499191279 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f50937fe700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50941027d0 0x7f5094197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:57.529 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5092ffd700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094105100 0x7f50941978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 -- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094105100 msgr2=0x7f50941978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094105100 0x7f50941978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 -- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50941027d0 msgr2=0x7f5094197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50941027d0 0x7f5094197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.526+0000 7f5093fff700 1 -- 172.21.15.146:0/2499191279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50840095f0 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5093fff700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f508c00b9e0 tx=0x7f508c00bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c00d8e0 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f508c009880 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c015e40 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509419f480 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.527+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509419f9b0 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.528+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50940fdda0 con 0x7f5094107a30 2024-02-10T05:56:57.530 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.528+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f508c00da40 con 0x7f5094107a30 2024-02-10T05:56:57.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.529+0000 7f5090ff9700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 0x7f507c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:56:57.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.529+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f508c011070 con 0x7f5094107a30 2024-02-10T05:56:57.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.529+0000 7f50937fe700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 0x7f507c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:56:57.531 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.530+0000 7f50937fe700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 0x7f507c06fe90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f508400b5c0 tx=0x7f5084005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:56:57.534 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.534+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f508c04fd20 con 0x7f5094107a30 2024-02-10T05:56:57.738 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:56:57.737+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5094062160 con 0x7f507c06d9d0 2024-02-10T05:56:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[41707]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[30020]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:58.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:58.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:58 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:58.343 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:58 smithi146 ceph-mon[33795]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:58.343 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:56:58.343 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:56:58.343 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:58 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:59.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:56:59 smithi121 ceph-mon[30020]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:56:59 smithi121 ceph-mon[41707]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:56:59.342 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:56:59 smithi146 ceph-mon[33795]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/2608954109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]': finished 2024-02-10T05:57:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[30020]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:57:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/2608954109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]': finished 2024-02-10T05:57:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[41707]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:57:00.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:00 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:00.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/2608954109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]: dispatch 2024-02-10T05:57:00.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27b049ad-0318-4757-bb5b-54c3835d24a9"}]': finished 2024-02-10T05:57:00.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:00 smithi146 ceph-mon[33795]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:57:00.860 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:00 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:01 smithi121 ceph-mon[30020]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/109231645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:01 smithi121 ceph-mon[41707]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/109231645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:01.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:01 smithi146 ceph-mon[33795]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:01.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/109231645' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:57:03.992 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:03 smithi146 ceph-mon[33795]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:57:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:03 smithi121 ceph-mon[30020]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:57:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:03 smithi121 ceph-mon[41707]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-10T05:57:06.052 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:05 smithi146 ceph-mon[33795]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-02-10T05:57:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:05 smithi121 ceph-mon[30020]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-02-10T05:57:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:05 smithi121 ceph-mon[41707]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-02-10T05:57:07.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:07 smithi146 ceph-mon[33795]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:07.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:07 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:57:07.892 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:07 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[30020]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:57:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[41707]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:08.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:57:08.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:07 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:08.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:08 smithi146 ceph-mon[33795]: Deploying daemon osd.7 on smithi146 2024-02-10T05:57:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:08 smithi121 ceph-mon[30020]: Deploying daemon osd.7 on smithi146 2024-02-10T05:57:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:08 smithi121 ceph-mon[41707]: Deploying daemon osd.7 on smithi146 2024-02-10T05:57:10.061 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:09 smithi146 ceph-mon[33795]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:09 smithi121 ceph-mon[30020]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:09 smithi121 ceph-mon[41707]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:57:11.876 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:11 smithi146 ceph-mon[33795]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:11 smithi121 ceph-mon[30020]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:12.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:11 smithi121 ceph-mon[41707]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[30020]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[41707]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:14.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:13 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:14.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:13 smithi146 ceph-mon[33795]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:13 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:13 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:13 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[30020]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[41707]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:15 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:15 smithi146 ceph-mon[33795]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:15 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.704+0000 7f5090ff9700 1 -- 172.21.15.146:0/2499191279 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5094062160 con 0x7f507c06d9d0 2024-02-10T05:57:16.704 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 msgr2=0x7f507c06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 0x7f507c06fe90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f508400b5c0 tx=0x7f5084005fd0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 msgr2=0x7f509419e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f508c00b9e0 tx=0x7f508c00bcf0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 shutdown_connections 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5094107a30 0x7f509419e900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50941027d0 0x7f5094197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f507c06d9d0 0x7f507c06fe90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 --2- 172.21.15.146:0/2499191279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5094105100 0x7f50941978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 >> 172.21.15.146:0/2499191279 conn(0x7f50940fc220 msgr2=0x7f5094103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 shutdown_connections 2024-02-10T05:57:16.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:16.706+0000 7f5099cc2700 1 -- 172.21.15.146:0/2499191279 wait complete. 2024-02-10T05:57:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:16 smithi146 ceph-mon[33795]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:16 smithi146 ceph-mon[33795]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:16 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:16 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[30020]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[30020]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[41707]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[41707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:57:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:17.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:16 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:17.366 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T05:57:17.368 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T05:57:17.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:57:17.617 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:57:17.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:57:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:57:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:17 smithi146 ceph-mon[33795]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[41707]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:17.991 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:57:17.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: from='osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:17.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:57:17.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:17.992 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:17 smithi121 ceph-mon[30020]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-02-10T05:57:18.492 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T05:57:18.293+0000 7fd46d3f3700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T05:57:18.492 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T05:57:18.300+0000 7fd4651de700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:57:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:57:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T05:57:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[41707]: from='osd.7 ' entity='osd.7' 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:18 smithi121 ceph-mon[30020]: from='osd.7 ' entity='osd.7' 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: osdmap e54: 8 total, 7 up, 8 in 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:18 smithi146 ceph-mon[33795]: from='osd.7 ' entity='osd.7' 2024-02-10T05:57:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 -- 172.21.15.121:0/4164202103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec1027f0 msgr2=0x7fd8ec1090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/4164202103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec1027f0 0x7fd8ec1090a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400b0d0 tx=0x7fd8e400b3e0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 -- 172.21.15.121:0/4164202103 shutdown_connections 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/4164202103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec1027f0 0x7fd8ec1090a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/4164202103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 0x7fd8ec101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/4164202103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec101e50 0x7fd8ec1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 -- 172.21.15.121:0/4164202103 >> 172.21.15.121:0/4164202103 conn(0x7fd8ec0fc1f0 msgr2=0x7fd8ec0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 -- 172.21.15.121:0/4164202103 shutdown_connections 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.367+0000 7fd8f16bb700 1 -- 172.21.15.121:0/4164202103 wait complete. 2024-02-10T05:57:19.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.368+0000 7fd8f16bb700 1 Processor -- start 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.368+0000 7fd8f16bb700 1 -- start start 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.368+0000 7fd8f16bb700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 0x7fd8ec196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 0x7fd8ec19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec19de00 con 0x7fd8ec101e50 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec19df40 con 0x7fd8ec100c50 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec19e0d0 con 0x7fd8ec1027f0 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35862/0 (socket says 172.21.15.121:35862) 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 -- 172.21.15.121:0/1359066792 learned_addr learned my addr 172.21.15.121:0/1359066792 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8eb7fe700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 0x7fd8ec19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:19.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8eaffd700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 0x7fd8ec196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 -- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 msgr2=0x7fd8ec19d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 0x7fd8ec19d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 -- 172.21.15.121:0/1359066792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 msgr2=0x7fd8ec196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 0x7fd8ec196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 -- 172.21.15.121:0/1359066792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0009520 con 0x7fd8ec101e50 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8eb7fe700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 0x7fd8ec19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8ea7fc700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000eb60 tx=0x7fd8e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e000cbd0 con 0x7fd8ec101e50 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e4009d00 con 0x7fd8ec101e50 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd8e00044a0 con 0x7fd8ec101e50 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.369+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0003ce0 con 0x7fd8ec101e50 2024-02-10T05:57:19.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.370+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec069df0 con 0x7fd8ec101e50 2024-02-10T05:57:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.371+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec0689e0 con 0x7fd8ec101e50 2024-02-10T05:57:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.371+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7fd8e0003e40 con 0x7fd8ec101e50 2024-02-10T05:57:19.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.372+0000 7fd8d3fff700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 0x7fd8d406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:19.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.372+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5320+0+0 (secure 0 0 0) 0x7fd8e0014070 con 0x7fd8ec101e50 2024-02-10T05:57:19.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.372+0000 7fd8eaffd700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 0x7fd8d406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:19.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.372+0000 7fd8eaffd700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 0x7fd8d406fe90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec101cb0 tx=0x7fd8dc007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:19.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.377+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8e0056e60 con 0x7fd8ec101e50 2024-02-10T05:57:19.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.568+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd8ec10bba0 con 0x7fd8ec101e50 2024-02-10T05:57:19.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.569+0000 7fd8d3fff700 1 -- 172.21.15.121:0/1359066792 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v56) v1 ==== 74+0+130 (secure 0 0 0) 0x7fd8e005a520 con 0x7fd8ec101e50 2024-02-10T05:57:19.569 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:57:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.570+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 msgr2=0x7fd8d406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.570+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 0x7fd8d406fe90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec101cb0 tx=0x7fd8dc007400 comp rx=0 tx=0).stop 2024-02-10T05:57:19.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.570+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 msgr2=0x7fd8ec1967b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.570+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000eb60 tx=0x7fd8e000ee70 comp rx=0 tx=0).stop 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 shutdown_connections 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd8d406d9d0 0x7fd8d406fe90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8ec1027f0 0x7fd8ec19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8ec100c50 0x7fd8ec196270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 --2- 172.21.15.121:0/1359066792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8ec101e50 0x7fd8ec1967b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 >> 172.21.15.121:0/1359066792 conn(0x7fd8ec0fc1f0 msgr2=0x7fd8ec0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 shutdown_connections 2024-02-10T05:57:19.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:19.571+0000 7fd8f16bb700 1 -- 172.21.15.121:0/1359066792 wait complete. 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: purged_snaps scrub starts 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: purged_snaps scrub ok 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] boot 2024-02-10T05:57:20.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:57:20.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.157 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1359066792' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: purged_snaps scrub starts 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: purged_snaps scrub ok 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] boot 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.160 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1359066792' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: purged_snaps scrub starts 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: purged_snaps scrub ok 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: osd.7 [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] boot 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:57:20.161 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1359066792' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:57:20.189 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1707544639,"num_in_osds":8,"osd_in_since":1707544620,"num_remapped_pgs":1} 2024-02-10T05:57:20.189 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:57:20.435 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:57:21.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: Detected new or changed devices on smithi146 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 4272M 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:21 smithi146 ceph-mon[33795]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:57:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: Detected new or changed devices on smithi146 2024-02-10T05:57:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 4272M 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:21.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[41707]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: Detected new or changed devices on smithi146 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 4272M 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:21.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:21 smithi121 ceph-mon[30020]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 -- 172.21.15.121:0/776690952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194100eb0 msgr2=0x7f2194101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 --2- 172.21.15.121:0/776690952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194100eb0 0x7f2194101290 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f217c009990 tx=0x7f217c009ca0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 -- 172.21.15.121:0/776690952 shutdown_connections 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 --2- 172.21.15.121:0/776690952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194105f00 0x7f219410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 --2- 172.21.15.121:0/776690952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194100eb0 0x7f2194101290 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 --2- 172.21.15.121:0/776690952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194101860 0x7f2194105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 -- 172.21.15.121:0/776690952 >> 172.21.15.121:0/776690952 conn(0x7f2194076c70 msgr2=0x7f2194077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 -- 172.21.15.121:0/776690952 shutdown_connections 2024-02-10T05:57:22.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.114+0000 7f219994c700 1 -- 172.21.15.121:0/776690952 wait complete. 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 Processor -- start 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 -- start start 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 0x7f219419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 0x7f21941976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194197e70 con 0x7f2194100eb0 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2194197fe0 con 0x7f2194101860 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f219994c700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21941981a0 con 0x7f2194105f00 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f2192ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f2192ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35878/0 (socket says 172.21.15.121:35878) 2024-02-10T05:57:22.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f2192ffd700 1 -- 172.21.15.121:0/104012572 learned_addr learned my addr 172.21.15.121:0/104012572 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f21937fe700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 0x7f21941976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.115+0000 7f21927fc700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 0x7f219419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 -- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 msgr2=0x7f21941976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 0x7f21941976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 -- 172.21.15.121:0/104012572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 msgr2=0x7f219419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 0x7f219419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 -- 172.21.15.121:0/104012572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184009520 con 0x7f2194100eb0 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f21937fe700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 0x7f21941976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f21927fc700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 0x7f219419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f2192ffd700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f217c004730 tx=0x7f217c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:22.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c01d070 con 0x7f2194100eb0 2024-02-10T05:57:22.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f217c00be20 con 0x7f2194100eb0 2024-02-10T05:57:22.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c0095f0 con 0x7f2194100eb0 2024-02-10T05:57:22.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c005290 con 0x7f2194100eb0 2024-02-10T05:57:22.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.116+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21941a43a0 con 0x7f2194100eb0 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.117+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2194102300 con 0x7f2194100eb0 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.117+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f217c00b9a0 con 0x7f2194100eb0 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.118+0000 7f219894a700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 0x7f218006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.118+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5316+0+0 (secure 0 0 0) 0x7f217c08dff0 con 0x7f2194100eb0 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.118+0000 7f21927fc700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 0x7f218006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:22.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.118+0000 7f21927fc700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 0x7f218006fe90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2194198e80 tx=0x7f2184009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:22.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.122+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f217c057db0 con 0x7f2194100eb0 2024-02-10T05:57:22.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.319+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2194198c50 con 0x7f2194100eb0 2024-02-10T05:57:22.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.320+0000 7f219894a700 1 -- 172.21.15.121:0/104012572 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13614 (secure 0 0 0) 0x7f217c05b470 con 0x7f2194100eb0 2024-02-10T05:57:22.321 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:57:22.321 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":58,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:51:53.237116+0000","modified":"2024-02-10T05:57:21.162609+0000","last_up_change":"2024-02-10T05:57:19.157828+0000","last_in_change":"2024-02-10T05:57:00.517452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:55:13.005730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e9f036-b22d-4b48-a265-3413e9eda4f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6803","nonce":889004683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6805","nonce":889004683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6809","nonce":889004683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6807","nonce":889004683}]},"public_addr":"172.21.15.121:6803/889004683","cluster_addr":"172.21.15.121:6805/889004683","heartbeat_back_addr":"172.21.15.121:6809/889004683","heartbeat_front_addr":"172.21.15.121:6807/889004683","state":["exists","up"]},{"osd":1,"uuid":"f733eda4-d0b0-4b09-84f2-5ec618b890db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6811","nonce":2658473683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6813","nonce":2658473683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6817","nonce":2658473683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6815","nonce":2658473683}]},"public_addr":"172.21.15.121:6811/2658473683","cluster_addr":"172.21.15.121:6813/2658473683","heartbeat_back_addr":"172.21.15.121:6817/2658473683","heartbeat_front_addr":"172.21.15.121:6815/2658473683","state":["exists","up"]},{"osd":2,"uuid":"c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6819","nonce":3662379322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6821","nonce":3662379322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6825","nonce":3662379322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6823","nonce":3662379322}]},"public_addr":"172.21.15.121:6819/3662379322","cluster_addr":"172.21.15.121:6821/3662379322","heartbeat_back_addr":"172.21.15.121:6825/3662379322","heartbeat_front_addr":"172.21.15.121:6823/3662379322","state":["exists","up"]},{"osd":3,"uuid":"b613b780-0fb2-4edf-b40f-2bc3c5782e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6827","nonce":255355367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6829","nonce":255355367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6833","nonce":255355367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6831","nonce":255355367}]},"public_addr":"172.21.15.121:6827/255355367","cluster_addr":"172.21.15.121:6829/255355367","heartbeat_back_addr":"172.21.15.121:6833/255355367","heartbeat_front_addr":"172.21.15.121:6831/255355367","state":["exists","up"]},{"osd":4,"uuid":"3bf95a4e-2fe0-4799-8e53-e730e7f49b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6801","nonce":855619125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6803","nonce":855619125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6807","nonce":855619125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6805","nonce":855619125}]},"public_addr":"172.21.15.146:6801/855619125","cluster_addr":"172.21.15.146:6803/855619125","heartbeat_back_addr":"172.21.15.146:6807/855619125","heartbeat_front_addr":"172.21.15.146:6805/855619125","state":["exists","up"]},{"osd":5,"uuid":"55de8feb-ec6c-4512-b30a-2275e97e8509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6809","nonce":3638337227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6811","nonce":3638337227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6815","nonce":3638337227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6813","nonce":3638337227}]},"public_addr":"172.21.15.146:6809/3638337227","cluster_addr":"172.21.15.146:6811/3638337227","heartbeat_back_addr":"172.21.15.146:6815/3638337227","heartbeat_front_addr":"172.21.15.146:6813/3638337227","state":["exists","up"]},{"osd":6,"uuid":"c45f670b-7a69-4f2a-98ae-ce1fba25a540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3393081156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3393081156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6823","nonce":3393081156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6821","nonce":3393081156}]},"public_addr":"172.21.15.146:6817/3393081156","cluster_addr":"172.21.15.146:6819/3393081156","heartbeat_back_addr":"172.21.15.146:6823/3393081156","heartbeat_front_addr":"172.21.15.146:6821/3393081156","state":["exists","up"]},{"osd":7,"uuid":"27b049ad-0318-4757-bb5b-54c3835d24a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6825","nonce":141569442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6827","nonce":141569442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6831","nonce":141569442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6829","nonce":141569442}]},"public_addr":"172.21.15.146:6825/141569442","cluster_addr":"172.21.15.146:6827/141569442","heartbeat_back_addr":"172.21.15.146:6831/141569442","heartbeat_front_addr":"172.21.15.146:6829/141569442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:18.533343+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:43.864363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:09.070052+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:34.771818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:58.601902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:24.563540+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.889604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:16.984463+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.121:0/92547055":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6801/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:0/3053598596":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2864864625":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2001688836":"2024-02-11T05:52:25.703724+0000","172.21.15.121:6801/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/3485345402":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/1796687448":"2024-02-11T05:52:52.671742+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:57:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 msgr2=0x7f218006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 0x7f218006fe90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2194198e80 tx=0x7f2184009260 comp rx=0 tx=0).stop 2024-02-10T05:57:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 msgr2=0x7f219419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f217c004730 tx=0x7f217c004760 comp rx=0 tx=0).stop 2024-02-10T05:57:22.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 shutdown_connections 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f218006d9d0 0x7f218006fe90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2194105f00 0x7f21941976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2194100eb0 0x7f219419d5f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 --2- 172.21.15.121:0/104012572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2194101860 0x7f219419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 >> 172.21.15.121:0/104012572 conn(0x7f2194076c70 msgr2=0x7f219406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 shutdown_connections 2024-02-10T05:57:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:22.322+0000 7f219994c700 1 -- 172.21.15.121:0/104012572 wait complete. 2024-02-10T05:57:22.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:22 smithi146 ceph-mon[33795]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:57:22.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:22 smithi121 ceph-mon[41707]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:57:22.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:22 smithi121 ceph-mon[30020]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:57:22.914 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T05:55:13.005730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-10T05:57:22.914 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T05:57:23.152 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/104012572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[30020]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[30020]: pgmap v184: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/104012572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[41707]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:57:23.519 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:23 smithi121 ceph-mon[41707]: pgmap v184: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/104012572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:57:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:23 smithi146 ceph-mon[33795]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:57:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:23 smithi146 ceph-mon[33795]: pgmap v184: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:24.427 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:24 smithi121 ceph-mon[30020]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:57:24.428 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:24 smithi121 ceph-mon[30020]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:57:24.428 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:24 smithi121 ceph-mon[41707]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:57:24.428 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:24 smithi121 ceph-mon[41707]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 -- 172.21.15.121:0/1351059575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06ae70 msgr2=0x7f3c7c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 --2- 172.21.15.121:0/1351059575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06ae70 0x7f3c7c106950 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009990 tx=0x7f3c70009ca0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 -- 172.21.15.121:0/1351059575 shutdown_connections 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 --2- 172.21.15.121:0/1351059575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c107030 0x7f3c7c10ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 --2- 172.21.15.121:0/1351059575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c06a4c0 0x7f3c7c06a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 --2- 172.21.15.121:0/1351059575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06ae70 0x7f3c7c106950 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 -- 172.21.15.121:0/1351059575 >> 172.21.15.121:0/1351059575 conn(0x7f3c7c076c70 msgr2=0x7f3c7c077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:24.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 -- 172.21.15.121:0/1351059575 shutdown_connections 2024-02-10T05:57:24.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.666+0000 7f3c82563700 1 -- 172.21.15.121:0/1351059575 wait complete. 2024-02-10T05:57:24.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 Processor -- start 2024-02-10T05:57:24.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 -- start start 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c06ae70 0x7f3c7c197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 0x7f3c7c19c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19c630 con 0x7f3c7c107030 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19c7a0 con 0x7f3c7c06ae70 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c82563700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c19c960 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53062/0 (socket says 172.21.15.121:53062) 2024-02-10T05:57:24.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 -- 172.21.15.121:0/333731902 learned_addr learned my addr 172.21.15.121:0/333731902 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c80b00700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 0x7f3c7c19c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7b7fe700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c06ae70 0x7f3c7c197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 -- 172.21.15.121:0/333731902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c06ae70 msgr2=0x7f3c7c197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c06ae70 0x7f3c7c197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 -- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 msgr2=0x7f3c7c19c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.667+0000 7f3c7bfff700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 0x7f3c7c19c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c7bfff700 1 -- 172.21.15.121:0/333731902 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c009520 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c80b00700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 0x7f3c7c19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c7bfff700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00eb30 tx=0x7f3c6c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c009880 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c700095f0 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3c6c0044a0 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c003d10 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.668+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c1a42d0 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.669+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68005120 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.669+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f3c6c00cc90 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.670+0000 7f3c797fa700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 0x7f3c6406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.670+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f3c6c014070 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.670+0000 7f3c7b7fe700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 0x7f3c6406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:24 smithi146 ceph-mon[33795]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-02-10T05:57:24.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:24 smithi146 ceph-mon[33795]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:57:24.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.670+0000 7f3c7b7fe700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 0x7f3c6406fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c70006010 tx=0x7f3c70005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:24.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.674+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3c6c056e30 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.868+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c68005950 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.868+0000 7f3c797fa700 1 -- 172.21.15.121:0/333731902 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c6c05a4f0 con 0x7f3c7c06a4c0 2024-02-10T05:57:24.869 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 msgr2=0x7f3c6406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 0x7f3c6406fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c70006010 tx=0x7f3c70005940 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 msgr2=0x7f3c7c1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00eb30 tx=0x7f3c6c00ee40 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 shutdown_connections 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c7c107030 0x7f3c7c19c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f3c6406d9d0 0x7f3c6406fe90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c7c06a4c0 0x7f3c7c1977a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 --2- 172.21.15.121:0/333731902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c7c06ae70 0x7f3c7c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 >> 172.21.15.121:0/333731902 conn(0x7f3c7c076c70 msgr2=0x7f3c7c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 shutdown_connections 2024-02-10T05:57:24.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:24.870+0000 7f3c82563700 1 -- 172.21.15.121:0/333731902 wait complete. 2024-02-10T05:57:25.524 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:25 smithi121 ceph-mon[30020]: pgmap v186: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:25.524 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/333731902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:57:25.524 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:25 smithi121 ceph-mon[41707]: pgmap v186: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:25.524 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/333731902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:57:25.553 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi121 2024-02-10T05:57:25.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi121=foo.a' 2024-02-10T05:57:25.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:25 smithi146 ceph-mon[33795]: pgmap v186: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-02-10T05:57:25.584 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/333731902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:57:25.807 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:26.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:26 smithi146 ceph-mon[33795]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-10T05:57:26.920 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:26 smithi146 ceph-mon[33795]: Cluster is now healthy 2024-02-10T05:57:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:26 smithi121 ceph-mon[30020]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-10T05:57:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:26 smithi121 ceph-mon[30020]: Cluster is now healthy 2024-02-10T05:57:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:26 smithi121 ceph-mon[41707]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-02-10T05:57:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:26 smithi121 ceph-mon[41707]: Cluster is now healthy 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- 172.21.15.146:0/2669232847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484107ab0 msgr2=0x7f5484109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 --2- 172.21.15.146:0/2669232847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484107ab0 0x7f5484109ee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f547c00b0a0 tx=0x7f547c00b3b0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- 172.21.15.146:0/2669232847 shutdown_connections 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 --2- 172.21.15.146:0/2669232847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484107ab0 0x7f5484109ee0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 --2- 172.21.15.146:0/2669232847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484102850 0x7f5484104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 --2- 172.21.15.146:0/2669232847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484105180 0x7f5484107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- 172.21.15.146:0/2669232847 >> 172.21.15.146:0/2669232847 conn(0x7f54840fc240 msgr2=0x7f54840fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- 172.21.15.146:0/2669232847 shutdown_connections 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- 172.21.15.146:0/2669232847 wait complete. 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 Processor -- start 2024-02-10T05:57:27.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.452+0000 7f548beb4700 1 -- start start 2024-02-10T05:57:27.455 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 0x7f54841973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 0x7f548419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548419ef20 con 0x7f5484102850 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548419f060 con 0x7f5484105180 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548419f1f0 con 0x7f5484107ab0 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46984/0 (socket says 172.21.15.146:46984) 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 -- 172.21.15.146:0/3670921299 learned_addr learned my addr 172.21.15.146:0/3670921299 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548a451700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 0x7f548419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f5489c50700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 0x7f54841973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 -- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 msgr2=0x7f548419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:27.456 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 0x7f548419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 -- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 msgr2=0x7f54841973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 0x7f54841973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 -- 172.21.15.146:0/3670921299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c009d00 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f5489c50700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 0x7f54841973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548a451700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 0x7f548419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548944f700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f547400d950 tx=0x7f547400dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474009b40 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5474009ca0 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547400f620 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.453+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548419f4d0 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.454+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548419f980 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.454+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54840fddc0 con 0x7f5484105180 2024-02-10T05:57:27.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.455+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f5474010460 con 0x7f5484105180 2024-02-10T05:57:27.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.455+0000 7f547affd700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 0x7f547006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:27.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.455+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f547408ce40 con 0x7f5484105180 2024-02-10T05:57:27.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.455+0000 7f5489c50700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 0x7f547006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:27.458 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.456+0000 7f5489c50700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 0x7f547006fe90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5480005950 tx=0x7f54800058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:27.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.460+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5474056b80 con 0x7f5484105180 2024-02-10T05:57:27.662 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.661+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi121=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f5484062160 con 0x7f547006d9d0 2024-02-10T05:57:27.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.668+0000 7f547affd700 1 -- 172.21.15.146:0/3670921299 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f5484062160 con 0x7f547006d9d0 2024-02-10T05:57:27.669 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled rgw.foo.a update... 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 msgr2=0x7f547006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 0x7f547006fe90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5480005950 tx=0x7f54800058e0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 msgr2=0x7f5484197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f547400d950 tx=0x7f547400dd10 comp rx=0 tx=0).stop 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 shutdown_connections 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.670+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f547006d9d0 0x7f547006fe90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5484107ab0 0x7f548419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5484102850 0x7f54841973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 --2- 172.21.15.146:0/3670921299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5484105180 0x7f5484197920 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:27.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 >> 172.21.15.146:0/3670921299 conn(0x7f54840fc240 msgr2=0x7f54841034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:27.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 shutdown_connections 2024-02-10T05:57:27.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:27.671+0000 7f548beb4700 1 -- 172.21.15.146:0/3670921299 wait complete. 2024-02-10T05:57:27.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:27 smithi146 ceph-mon[33795]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:27.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:27 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:27 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:28.307 DEBUG:teuthology.orchestra.run.smithi121:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service 2024-02-10T05:57:28.309 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi146 2024-02-10T05:57:28.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-10T05:57:28.327 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[41707]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[30020]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T05:57:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:27 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:28.587 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:28.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi121=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:28.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:28 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi121=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:29.088 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi121=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:29.089 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:28 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:30.031 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:29 smithi121 ceph-mon[41707]: Deploying daemon rgw.foo.a on smithi121 2024-02-10T05:57:30.031 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:29 smithi121 ceph-mon[41707]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:57:30.032 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:29 smithi121 ceph-mon[30020]: Deploying daemon rgw.foo.a on smithi121 2024-02-10T05:57:30.032 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:29 smithi121 ceph-mon[30020]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:57:30.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:29 smithi146 ceph-mon[33795]: Deploying daemon rgw.foo.a on smithi121 2024-02-10T05:57:30.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:29 smithi146 ceph-mon[33795]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 -- 172.21.15.146:0/2038290354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488102880 msgr2=0x7fb488104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/2038290354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488102880 0x7fb488104c70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb47c009990 tx=0x7fb47c009ca0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 -- 172.21.15.146:0/2038290354 shutdown_connections 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/2038290354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb488107ae0 0x7fb488109f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/2038290354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488102880 0x7fb488104c70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/2038290354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4881051b0 0x7fb4881075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 -- 172.21.15.146:0/2038290354 >> 172.21.15.146:0/2038290354 conn(0x7fb4880fc2b0 msgr2=0x7fb4880fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 -- 172.21.15.146:0/2038290354 shutdown_connections 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.320+0000 7fb48e1d5700 1 -- 172.21.15.146:0/2038290354 wait complete. 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 Processor -- start 2024-02-10T05:57:30.337 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 -- start start 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 0x7fb488197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48819edf0 con 0x7fb4881051b0 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48819ef30 con 0x7fb488107ae0 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb48e1d5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48819f0c0 con 0x7fb488102880 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb487fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb486ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb4877fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 0x7fb488197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb486ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.146:33608/0 (socket says 172.21.15.146:33608) 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb487fff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:47010/0 (socket says 172.21.15.146:47010) 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.321+0000 7fb486ffd700 1 -- 172.21.15.146:0/750630991 learned_addr learned my addr 172.21.15.146:0/750630991 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:30.338 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 -- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 msgr2=0x7fb488197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 0x7fb488197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 -- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 msgr2=0x7fb4881977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 -- 172.21.15.146:0/750630991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0095f0 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb4877fe700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 0x7fb488197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb486ffd700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb487fff700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb48000ef90 tx=0x7fb48000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480018070 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48819f340 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb48000f040 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48000cba0 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.322+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48819f840 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.323+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4880fde30 con 0x7fb488107ae0 2024-02-10T05:57:30.339 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.324+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7fb48000cd20 con 0x7fb488107ae0 2024-02-10T05:57:30.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.324+0000 7fb484ff9700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 0x7fb47006fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:30.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.324+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fb48008cbb0 con 0x7fb488107ae0 2024-02-10T05:57:30.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.324+0000 7fb4877fe700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 0x7fb47006fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:30.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.324+0000 7fb4877fe700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 0x7fb47006fe90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00b2e0 tx=0x7fb47c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:30.340 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.329+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb480056870 con 0x7fb488107ae0 2024-02-10T05:57:30.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.522+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fb48810cba0 con 0x7fb488107ae0 2024-02-10T05:57:30.578 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:30 smithi121 systemd[1]: Starting Ceph rgw.foo.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:30.846 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.845+0000 7fb484ff9700 1 -- 172.21.15.146:0/750630991 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v61) v1 ==== 160+0+0 (secure 0 0 0) 0x7fb480059f30 con 0x7fb488107ae0 2024-02-10T05:57:30.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 msgr2=0x7fb47006fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:30.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 0x7fb47006fe90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00b2e0 tx=0x7fb47c005fb0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.848 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 msgr2=0x7fb48819e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb48000ef90 tx=0x7fb48000c3a0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 shutdown_connections 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb488107ae0 0x7fb48819e820 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb47006d9d0 0x7fb47006fe90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb488102880 0x7fb488197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 --2- 172.21.15.146:0/750630991 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4881051b0 0x7fb4881977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 >> 172.21.15.146:0/750630991 conn(0x7fb4880fc2b0 msgr2=0x7fb4881034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 shutdown_connections 2024-02-10T05:57:30.849 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:30.848+0000 7fb48e1d5700 1 -- 172.21.15.146:0/750630991 wait complete. 2024-02-10T05:57:30.850 INFO:teuthology.orchestra.run.smithi146.stderr:pool 'datapool' created 2024-02-10T05:57:30.968 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/750630991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:30.968 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:30.968 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/750630991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:30.968 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:30.969 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:30 smithi121 podman[69649]: 2024-02-10 05:57:30.621013182 +0000 UTC m=+0.018418262 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:30.969 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:30 smithi121 podman[69649]: 2024-02-10 05:57:30.725514395 +0000 UTC m=+0.122919467 container create 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T05:57:30.969 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:30 smithi121 podman[69649]: 2024-02-10 05:57:30.962651865 +0000 UTC m=+0.360056939 container init 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T05:57:30.969 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:30 smithi121 podman[69649]: 2024-02-10 05:57:30.968226593 +0000 UTC m=+0.365631668 container start 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:57:31.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/750630991' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:31.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:57:31.328 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:31 smithi121 bash[69649]: 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 2024-02-10T05:57:31.328 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 05:57:31 smithi121 systemd[1]: Started Ceph rgw.foo.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:57:31.567 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- rbd pool init datapool 2024-02-10T05:57:31.804 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:32.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:31 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.223 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:32.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:31 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3930009905' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:57:33.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:32 smithi146 ceph-mon[33795]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3930009905' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:57:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[41707]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: Saving service rgw.foo.a spec with placement smithi121=foo.a;count:1 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3930009905' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:57:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:32 smithi121 ceph-mon[30020]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: pgmap v192: 36 pgs: 7 creating+peering, 2 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/2853908074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:57:34.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:57:34.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: pgmap v192: 36 pgs: 7 creating+peering, 2 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/2853908074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: pgmap v192: 36 pgs: 7 creating+peering, 2 active+clean, 27 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/2853908074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:57:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:57:34.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:57:34.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:34 smithi146 ceph-mon[33795]: Checking dashboard <-> RGW credentials 2024-02-10T05:57:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:34 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[30020]: Checking dashboard <-> RGW credentials 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[41707]: Checking dashboard <-> RGW credentials 2024-02-10T05:57:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:57:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:35 smithi146 ceph-mon[33795]: pgmap v195: 68 pgs: 7 creating+peering, 4 active+clean, 57 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:35 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:35 smithi146 ceph-mon[33795]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:57:36.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[30020]: pgmap v195: 68 pgs: 7 creating+peering, 4 active+clean, 57 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:36.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:36.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[30020]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:57:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[41707]: pgmap v195: 68 pgs: 7 creating+peering, 4 active+clean, 57 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:57:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:57:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:35 smithi121 ceph-mon[41707]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:57:36.580 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.146 --placement '1;smithi146=iscsi.a' 2024-02-10T05:57:36.820 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:37.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:36 smithi146 ceph-mon[33795]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:57:37.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:36 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:36 smithi146 ceph-mon[33795]: pgmap v198: 100 pgs: 18 creating+peering, 51 active+clean, 31 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[30020]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[30020]: pgmap v198: 100 pgs: 18 creating+peering, 51 active+clean, 31 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[41707]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:57:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:57:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:36 smithi121 ceph-mon[41707]: pgmap v198: 100 pgs: 18 creating+peering, 51 active+clean, 31 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:37 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:37 smithi146 ceph-mon[33795]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[41707]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:57:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:37 smithi121 ceph-mon[30020]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:57:38.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- 172.21.15.146:0/3922354886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 msgr2=0x7efefc102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 --2- 172.21.15.146:0/3922354886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc102190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efeec0099c0 tx=0x7efeec009cd0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- 172.21.15.146:0/3922354886 shutdown_connections 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 --2- 172.21.15.146:0/3922354886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 0x7efefc10a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 --2- 172.21.15.146:0/3922354886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc102190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 --2- 172.21.15.146:0/3922354886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 0x7efefc1033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- 172.21.15.146:0/3922354886 >> 172.21.15.146:0/3922354886 conn(0x7efefc0fd330 msgr2=0x7efefc0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- 172.21.15.146:0/3922354886 shutdown_connections 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- 172.21.15.146:0/3922354886 wait complete. 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 Processor -- start 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.483+0000 7eff0254f700 1 -- start start 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:38.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 0x7efefc074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 0x7efefc074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc074dd0 con 0x7efefc102f70 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc074f40 con 0x7efefc101d70 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff0254f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc1a1290 con 0x7efefc103910 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59860/0 (socket says 172.21.15.146:59860) 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 -- 172.21.15.146:0/2335676832 learned_addr learned my addr 172.21.15.146:0/2335676832 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefb7fe700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 0x7efefc074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7eff00aec700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 0x7efefc074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 -- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 msgr2=0x7efefc074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 0x7efefc074650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 -- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 msgr2=0x7efefc074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 0x7efefc074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 -- 172.21.15.146:0/2335676832 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec0095f0 con 0x7efefc101d70 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.484+0000 7efefbfff700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efeec009f50 tx=0x7efeec00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.485+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec01d070 con 0x7efefc101d70 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.485+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7efeec022470 con 0x7efefc101d70 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.485+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc1a1510 con 0x7efefc101d70 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.485+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec0037c0 con 0x7efefc101d70 2024-02-10T05:57:38.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.485+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc1a1a00 con 0x7efefc101d70 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.486+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efefc108950 con 0x7efefc101d70 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.486+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7efeec022890 con 0x7efefc101d70 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.486+0000 7efef97fa700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 0x7efee406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.486+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7121+0+0 (secure 0 0 0) 0x7efeec08ebe0 con 0x7efefc101d70 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.487+0000 7efefb7fe700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 0x7efee406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:38.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.487+0000 7efefb7fe700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 0x7efee406ff30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efef0005950 tx=0x7efef000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:38.491 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.491+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efeec0583c0 con 0x7efefc101d70 2024-02-10T05:57:38.695 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.695+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7efefc10cd20 con 0x7efee406da70 2024-02-10T05:57:38.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.705+0000 7efef97fa700 1 -- 172.21.15.146:0/2335676832 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7efefc10cd20 con 0x7efee406da70 2024-02-10T05:57:38.706 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled iscsi.datapool update... 2024-02-10T05:57:38.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 msgr2=0x7efee406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:38.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 0x7efee406ff30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efef0005950 tx=0x7efef000a300 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 msgr2=0x7efefc075ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efeec009f50 tx=0x7efeec00f690 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 shutdown_connections 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7efee406da70 0x7efee406ff30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efefc103910 0x7efefc074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efefc101d70 0x7efefc075ac0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 --2- 172.21.15.146:0/2335676832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efefc102f70 0x7efefc074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 >> 172.21.15.146:0/2335676832 conn(0x7efefc0fd330 msgr2=0x7efefc0fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 shutdown_connections 2024-02-10T05:57:38.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:38.708+0000 7eff0254f700 1 -- 172.21.15.146:0/2335676832 wait complete. 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:39.025 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: pgmap v201: 132 pgs: 12 creating+peering, 74 active+clean, 46 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.026 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:38 smithi146 ceph-mon[33795]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:57:39.310 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-10T05:57:39.310 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:57:39.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: pgmap v201: 132 pgs: 12 creating+peering, 74 active+clean, 46 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[30020]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:57:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.146", "placement": "1;smithi146=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: Saving service iscsi.datapool spec with placement smithi146=iscsi.a;count:1 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: pgmap v201: 132 pgs: 12 creating+peering, 74 active+clean, 46 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:57:39.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:38 smithi121 ceph-mon[41707]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:57:39.346 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:57:39.346 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T05:57:39.375 DEBUG:teuthology.orchestra.run.smithi146:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service 2024-02-10T05:57:39.417 INFO:tasks.cephadm:Adding prometheus.a on smithi146 2024-02-10T05:57:39.417 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi146=a' 2024-02-10T05:57:39.691 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:39 smithi146 ceph-mon[33795]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[30020]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1780533274' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2766397985' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:57:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:39 smithi121 ceph-mon[41707]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 -- 172.21.15.146:0/318921186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 msgr2=0x7f5a2c102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 --2- 172.21.15.146:0/318921186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c102160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400b0d0 tx=0x7f5a2400b3e0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 -- 172.21.15.146:0/318921186 shutdown_connections 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 --2- 172.21.15.146:0/318921186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a2c1038e0 0x7f5a2c10a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 --2- 172.21.15.146:0/318921186 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c102160 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 --2- 172.21.15.146:0/318921186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a2c102f40 0x7f5a2c1033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.326 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 -- 172.21.15.146:0/318921186 >> 172.21.15.146:0/318921186 conn(0x7f5a2c0fd300 msgr2=0x7f5a2c0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 -- 172.21.15.146:0/318921186 shutdown_connections 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.324+0000 7f5a3362b700 1 -- 172.21.15.146:0/318921186 wait complete. 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 Processor -- start 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 -- start start 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a2c102f40 0x7f5a2c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a2c1038e0 0x7f5a2c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c19f0d0 con 0x7f5a2c102f40 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c19f210 con 0x7f5a2c101d40 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a3362b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c19f3a0 con 0x7f5a2c1038e0 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59882/0 (socket says 172.21.15.146:59882) 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 -- 172.21.15.146:0/4275943761 learned_addr learned my addr 172.21.15.146:0/4275943761 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:41.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 -- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a2c1038e0 msgr2=0x7f5a2c19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a2c1038e0 0x7f5a2c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 -- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a2c102f40 msgr2=0x7f5a2c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a2c102f40 0x7f5a2c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 -- 172.21.15.146:0/4275943761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24009d00 con 0x7f5a2c101d40 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a313c7700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5a24007930 tx=0x7f5a24009180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.325+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2400e040 con 0x7f5a2c101d40 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.326+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c19f620 con 0x7f5a2c101d40 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.326+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5a24009490 con 0x7f5a2c101d40 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.326+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2401bb50 con 0x7f5a2c101d40 2024-02-10T05:57:41.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.326+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c19fb60 con 0x7f5a2c101d40 2024-02-10T05:57:41.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.327+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2c108920 con 0x7f5a2c101d40 2024-02-10T05:57:41.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.327+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f5a2401bcb0 con 0x7f5a2c101d40 2024-02-10T05:57:41.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.327+0000 7f5a2a7fc700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 0x7f5a1406fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:41.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.327+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f5a2408f1e0 con 0x7f5a2c101d40 2024-02-10T05:57:41.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.328+0000 7f5a30bc6700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 0x7f5a1406fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:41.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.328+0000 7f5a30bc6700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 0x7f5a1406fe90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a200097b0 tx=0x7f5a20006d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:41.334 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.333+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a24058760 con 0x7f5a2c101d40 2024-02-10T05:57:41.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: Deploying daemon iscsi.iscsi.a on smithi146 2024-02-10T05:57:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:41 smithi146 ceph-mon[33795]: pgmap v204: 132 pgs: 12 creating+peering, 99 active+clean, 21 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-02-10T05:57:41.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.553+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a2c062160 con 0x7f5a1406d9d0 2024-02-10T05:57:41.563 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.563+0000 7f5a2a7fc700 1 -- 172.21.15.146:0/4275943761 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f5a2c062160 con 0x7f5a1406d9d0 2024-02-10T05:57:41.563 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled prometheus update... 2024-02-10T05:57:41.565 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 msgr2=0x7f5a1406fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 0x7f5a1406fe90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5a200097b0 tx=0x7f5a20006d20 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 msgr2=0x7f5a2c197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5a24007930 tx=0x7f5a24009180 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 shutdown_connections 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f5a1406d9d0 0x7f5a1406fe90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a2c1038e0 0x7f5a2c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a2c101d40 0x7f5a2c197540 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 --2- 172.21.15.146:0/4275943761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a2c102f40 0x7f5a2c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 >> 172.21.15.146:0/4275943761 conn(0x7f5a2c0fd300 msgr2=0x7f5a2c0fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:41.566 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 shutdown_connections 2024-02-10T05:57:41.567 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:41.565+0000 7f5a3362b700 1 -- 172.21.15.146:0/4275943761 wait complete. 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:57:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: Deploying daemon iscsi.iscsi.a on smithi146 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[41707]: pgmap v204: 132 pgs: 12 creating+peering, 99 active+clean, 21 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:41.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: Deploying daemon iscsi.iscsi.a on smithi146 2024-02-10T05:57:41.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:41 smithi121 ceph-mon[30020]: pgmap v204: 132 pgs: 12 creating+peering, 99 active+clean, 21 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-02-10T05:57:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:42 smithi146 ceph-mon[33795]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:57:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:42 smithi146 ceph-mon[33795]: Cluster is now healthy 2024-02-10T05:57:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:42 smithi146 ceph-mon[33795]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:42 smithi146 ceph-mon[33795]: Saving service prometheus spec with placement smithi146=a;count:1 2024-02-10T05:57:42.418 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:42 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[30020]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:57:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[30020]: Cluster is now healthy 2024-02-10T05:57:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[30020]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[30020]: Saving service prometheus spec with placement smithi146=a;count:1 2024-02-10T05:57:42.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:42.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[41707]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:57:42.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[41707]: Cluster is now healthy 2024-02-10T05:57:42.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[41707]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:42.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[41707]: Saving service prometheus spec with placement smithi146=a;count:1 2024-02-10T05:57:42.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:42 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:42.731 DEBUG:teuthology.orchestra.run.smithi146:prometheus.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T05:57:42.734 INFO:tasks.cephadm:Adding node-exporter.a on smithi121 2024-02-10T05:57:42.734 INFO:tasks.cephadm:Adding node-exporter.b on smithi146 2024-02-10T05:57:42.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi121=a;smithi146=b' 2024-02-10T05:57:43.250 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:43.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:43 smithi146 ceph-mon[33795]: pgmap v205: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 5.7 KiB/s wr, 249 op/s 2024-02-10T05:57:43.407 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:43 smithi146 ceph-mon[33795]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:57:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:43 smithi121 ceph-mon[41707]: pgmap v205: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 5.7 KiB/s wr, 249 op/s 2024-02-10T05:57:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:43 smithi121 ceph-mon[41707]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:57:43.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:43 smithi121 ceph-mon[30020]: pgmap v205: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 5.7 KiB/s wr, 249 op/s 2024-02-10T05:57:43.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:43 smithi121 ceph-mon[30020]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:57:45.277 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 podman[63323]: 2024-02-10 05:57:45.041668356 +0000 UTC m=+0.796673900 container create a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True) 2024-02-10T05:57:45.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 -- 172.21.15.146:0/27255554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 msgr2=0x7f98dc109140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:45.386 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/27255554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc109140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b0a0 tx=0x7f98d400b3b0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 -- 172.21.15.146:0/27255554 shutdown_connections 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/27255554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc109140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/27255554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc102280 0x7f98dc1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/27255554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc100ee0 0x7f98dc101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 -- 172.21.15.146:0/27255554 >> 172.21.15.146:0/27255554 conn(0x7f98dc0fb970 msgr2=0x7f98dc0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 -- 172.21.15.146:0/27255554 shutdown_connections 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.383+0000 7f98e2ab7700 1 -- 172.21.15.146:0/27255554 wait complete. 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 Processor -- start 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 -- start start 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc100ee0 0x7f98dc1975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc102280 0x7f98dc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:45.387 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19f130 con 0x7f98dc102280 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19f270 con 0x7f98dc1018a0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e2ab7700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19f400 con 0x7f98dc100ee0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:45924/0 (socket says 172.21.15.146:45924) 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 -- 172.21.15.146:0/2326368213 learned_addr learned my addr 172.21.15.146:0/2326368213 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e1054700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc102280 0x7f98dc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98e0853700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc100ee0 0x7f98dc1975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 -- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc100ee0 msgr2=0x7f98dc1975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc100ee0 0x7f98dc1975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 -- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc102280 msgr2=0x7f98dc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc102280 0x7f98dc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.384+0000 7f98dbfff700 1 -- 172.21.15.146:0/2326368213 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4009d00 con 0x7f98dc1018a0 2024-02-10T05:57:45.388 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98dbfff700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f98c800b530 tx=0x7f98c800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c8011830 con 0x7f98dc1018a0 2024-02-10T05:57:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f98c8011e10 con 0x7f98dc1018a0 2024-02-10T05:57:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc19f6e0 con 0x7f98dc1018a0 2024-02-10T05:57:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c8010d80 con 0x7f98dc1018a0 2024-02-10T05:57:45.389 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.385+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc19fcb0 con 0x7f98dc1018a0 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.386+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc108ad0 con 0x7f98dc1018a0 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.386+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 15) v1 ==== 91407+0+0 (secure 0 0 0) 0x7f98c8011990 con 0x7f98dc1018a0 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.387+0000 7f98d9ffb700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 0x7f98cc06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.387+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f98c808d130 con 0x7f98dc1018a0 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.387+0000 7f98e0853700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 0x7f98cc06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:45.390 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.387+0000 7f98e0853700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 0x7f98cc06fe90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f98d0006fd0 tx=0x7f98d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:45.393 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.392+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98c8056730 con 0x7f98dc1018a0 2024-02-10T05:57:45.607 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.605+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi121=a;smithi146=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f98dc10be00 con 0x7f98cc06d9d0 2024-02-10T05:57:45.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 podman[63323]: 2024-02-10 05:57:45.486946452 +0000 UTC m=+1.241951942 container init a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T05:57:45.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 podman[63323]: 2024-02-10 05:57:45.494620865 +0000 UTC m=+1.249626341 container start a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131) 2024-02-10T05:57:45.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.612+0000 7f98d9ffb700 1 -- 172.21.15.146:0/2326368213 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f98dc10be00 con 0x7f98cc06d9d0 2024-02-10T05:57:45.614 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled node-exporter update... 2024-02-10T05:57:45.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 msgr2=0x7f98cc06fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:45.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 0x7f98cc06fe90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f98d0006fd0 tx=0x7f98d0008040 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 msgr2=0x7f98dc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f98c800b530 tx=0x7f98c800b8f0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 shutdown_connections 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc102280 0x7f98dc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f98cc06d9d0 0x7f98cc06fe90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc100ee0 0x7f98dc1975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 --2- 172.21.15.146:0/2326368213 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1018a0 0x7f98dc197ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 >> 172.21.15.146:0/2326368213 conn(0x7f98dc0fb970 msgr2=0x7f98dc100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 shutdown_connections 2024-02-10T05:57:45.618 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:45.616+0000 7f98e2ab7700 1 -- 172.21.15.146:0/2326368213 wait complete. 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:45.878 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:45.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 bash[63323]: a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa 2024-02-10T05:57:45.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 systemd[1]: Started Ceph iscsi.iscsi.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:57:45.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Started the configuration object watcher 2024-02-10T05:57:45.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Checking for config object changes every 1s 2024-02-10T05:57:45.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:45 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Processing osd blocklist entries for this node 2024-02-10T05:57:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-02-10T05:57:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:45 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:46.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Reading the configuration object to update local LIO configuration 2024-02-10T05:57:46.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Configuration does not have an entry for this host(smithi146.front.sepia.ceph.com) - nothing to define to LIO 2024-02-10T05:57:46.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-10T05:57:46.347 DEBUG:teuthology.orchestra.run.smithi121:node-exporter.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T05:57:46.350 DEBUG:teuthology.orchestra.run.smithi146:node-exporter.b> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T05:57:46.352 INFO:tasks.cephadm:Adding alertmanager.a on smithi121 2024-02-10T05:57:46.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi121=a' 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[30020]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi121=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[30020]: Saving service node-exporter spec with placement smithi121=a;smithi146=b;count:2 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[30020]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/3102329435' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:57:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[41707]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi121=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[41707]: Saving service node-exporter spec with placement smithi121=a;smithi146=b;count:2 2024-02-10T05:57:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[41707]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:57:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/3102329435' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:57:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-mon[33795]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi121=a;smithi146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-mon[33795]: Saving service node-exporter spec with placement smithi121=a;smithi146=b;count:2 2024-02-10T05:57:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-mon[33795]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:57:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:57:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/3102329435' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:57:47.285 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[41707]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.5 KiB/s wr, 204 op/s 2024-02-10T05:57:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[41707]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:57:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[30020]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.5 KiB/s wr, 204 op/s 2024-02-10T05:57:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[30020]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:57:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:47 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:48.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:47 smithi146 ceph-mon[33795]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 4.5 KiB/s wr, 204 op/s 2024-02-10T05:57:48.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:47 smithi146 ceph-mon[33795]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:57:48.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:47 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:49.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:49 smithi146 ceph-mon[33795]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-02-10T05:57:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:49 smithi121 ceph-mon[41707]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-02-10T05:57:49.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:49 smithi121 ceph-mon[30020]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 4.0 KiB/s wr, 181 op/s 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 -- 172.21.15.146:0/1774765504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 msgr2=0x7f1b281022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 --2- 172.21.15.146:0/1774765504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b281022a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b180099c0 tx=0x7f1b18009cd0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 -- 172.21.15.146:0/1774765504 shutdown_connections 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 --2- 172.21.15.146:0/1774765504 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b281027e0 0x7f1b28109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 --2- 172.21.15.146:0/1774765504 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b28100c40 0x7f1b28101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 --2- 172.21.15.146:0/1774765504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b281022a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 -- 172.21.15.146:0/1774765504 >> 172.21.15.146:0/1774765504 conn(0x7f1b280fc1e0 msgr2=0x7f1b280fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 -- 172.21.15.146:0/1774765504 shutdown_connections 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.412+0000 7f1b2d006700 1 -- 172.21.15.146:0/1774765504 wait complete. 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 Processor -- start 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 -- start start 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b28100c40 0x7f1b2810b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b281027e0 0x7f1b2810a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2810a860 con 0x7f1b28100c40 2024-02-10T05:57:49.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2810a9d0 con 0x7f1b28101e40 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2d006700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b281a1340 con 0x7f1b281027e0 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46010/0 (socket says 172.21.15.146:46010) 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 -- 172.21.15.146:0/2440552247 learned_addr learned my addr 172.21.15.146:0/2440552247 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b26d9d700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b28100c40 0x7f1b2810b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2759e700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b281027e0 0x7f1b2810a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 -- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b281027e0 msgr2=0x7f1b2810a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b281027e0 0x7f1b2810a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 -- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b28100c40 msgr2=0x7f1b2810b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b28100c40 0x7f1b2810b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.413+0000 7f1b2659c700 1 -- 172.21.15.146:0/2440552247 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b180095f0 con 0x7f1b28101e40 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b2659c700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1b18005910 tx=0x7f1b18005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:49.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1801d070 con 0x7f1b28101e40 2024-02-10T05:57:49.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b281a15c0 con 0x7f1b28101e40 2024-02-10T05:57:49.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1b1800bd10 con 0x7f1b28101e40 2024-02-10T05:57:49.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b18022c30 con 0x7f1b28101e40 2024-02-10T05:57:49.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.414+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b281a1a30 con 0x7f1b28101e40 2024-02-10T05:57:49.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.415+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b281080d0 con 0x7f1b28101e40 2024-02-10T05:57:49.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.415+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7f1b180037c0 con 0x7f1b28101e40 2024-02-10T05:57:49.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.416+0000 7f1b1ffff700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 0x7f1b1406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:49.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.416+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f1b1808e8e0 con 0x7f1b28101e40 2024-02-10T05:57:49.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.416+0000 7f1b26d9d700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 0x7f1b1406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:49.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.416+0000 7f1b26d9d700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 0x7f1b1406ff20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1b28101ca0 tx=0x7f1b1000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:49.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.421+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b18057d60 con 0x7f1b28101e40 2024-02-10T05:57:49.640 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.639+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b28062160 con 0x7f1b1406da60 2024-02-10T05:57:49.647 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.646+0000 7f1b1ffff700 1 -- 172.21.15.146:0/2440552247 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f1b28062160 con 0x7f1b1406da60 2024-02-10T05:57:49.647 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled alertmanager update... 2024-02-10T05:57:49.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 msgr2=0x7f1b1406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:49.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 0x7f1b1406ff20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1b28101ca0 tx=0x7f1b1000b410 comp rx=0 tx=0).stop 2024-02-10T05:57:49.649 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 msgr2=0x7f1b28109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1b18005910 tx=0x7f1b18005b00 comp rx=0 tx=0).stop 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 shutdown_connections 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f1b1406da60 0x7f1b1406ff20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b281027e0 0x7f1b2810a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b28100c40 0x7f1b2810b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 --2- 172.21.15.146:0/2440552247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b28101e40 0x7f1b28109ba0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:49.650 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 >> 172.21.15.146:0/2440552247 conn(0x7f1b280fc1e0 msgr2=0x7f1b28105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:49.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 shutdown_connections 2024-02-10T05:57:49.651 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:49.649+0000 7f1b2d006700 1 -- 172.21.15.146:0/2440552247 wait complete. 2024-02-10T05:57:50.434 DEBUG:teuthology.orchestra.run.smithi121:alertmanager.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T05:57:50.437 INFO:tasks.cephadm:Adding grafana.a on smithi146 2024-02-10T05:57:50.438 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi146=a' 2024-02-10T05:57:50.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:50 smithi146 ceph-mon[33795]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:50 smithi146 ceph-mon[33795]: Saving service alertmanager spec with placement smithi121=a;count:1 2024-02-10T05:57:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:50 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:50.823 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[41707]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[41707]: Saving service alertmanager spec with placement smithi121=a;count:1 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[30020]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi121=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[30020]: Saving service alertmanager spec with placement smithi121=a;count:1 2024-02-10T05:57:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:50 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.8 KiB/s wr, 121 op/s 2024-02-10T05:57:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:51 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.8 KiB/s wr, 121 op/s 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.8 KiB/s wr, 121 op/s 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:57:52.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:51 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/1064534062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4100420 msgr2=0x7fd9a4100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/1064534062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4100420 0x7fd9a4100840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd9940099c0 tx=0x7fd994009cd0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/1064534062 shutdown_connections 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/1064534062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4101fc0 0x7fd9a4108870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/1064534062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4100420 0x7fd9a4100840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/1064534062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101620 0x7fd9a4101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/1064534062 >> 172.21.15.146:0/1064534062 conn(0x7fd9a40fb9c0 msgr2=0x7fd9a40fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/1064534062 shutdown_connections 2024-02-10T05:57:52.516 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/1064534062 wait complete. 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 Processor -- start 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.514+0000 7fd9a9f4a700 1 -- start start 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 0x7fd9a4197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101fc0 0x7fd9a419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a419eeb0 con 0x7fd9a4101fc0 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a419eff0 con 0x7fd9a4101620 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a419f180 con 0x7fd9a4100420 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:46036/0 (socket says 172.21.15.146:46036) 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 -- 172.21.15.146:0/701417482 learned_addr learned my addr 172.21.15.146:0/701417482 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a3fff700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101fc0 0x7fd9a419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a37fe700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 0x7fd9a4197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:52.517 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 -- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 msgr2=0x7fd9a4197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 0x7fd9a4197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 -- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101fc0 msgr2=0x7fd9a419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101fc0 0x7fd9a419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 -- 172.21.15.146:0/701417482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940095f0 con 0x7fd9a4101620 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a37fe700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 0x7fd9a4197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd99affd700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00b530 tx=0x7fd98c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c011830 con 0x7fd9a4101620 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd98c011e10 con 0x7fd9a4101620 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c010e60 con 0x7fd9a4101620 2024-02-10T05:57:52.518 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.515+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9a419f460 con 0x7fd9a4101620 2024-02-10T05:57:52.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.516+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9a419f9e0 con 0x7fd9a4101620 2024-02-10T05:57:52.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.516+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9a4108200 con 0x7fd9a4101620 2024-02-10T05:57:52.519 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.517+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7fd98c011990 con 0x7fd9a4101620 2024-02-10T05:57:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.518+0000 7fd9a17fa700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 0x7fd99006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.518+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7148+0+0 (secure 0 0 0) 0x7fd98c08d210 con 0x7fd9a4101620 2024-02-10T05:57:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.518+0000 7fd9a37fe700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 0x7fd99006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:52.520 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.518+0000 7fd9a37fe700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 0x7fd99006ff20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd994000c00 tx=0x7fd99400bfb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:52.523 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.523+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd98c0567a0 con 0x7fd9a4101620 2024-02-10T05:57:52.732 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.731+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 --> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9a4062160 con 0x7fd99006da60 2024-02-10T05:57:52.739 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.739+0000 7fd9a17fa700 1 -- 172.21.15.146:0/701417482 <== mgr.14150 v2:172.21.15.121:6800/2329816948 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fd9a4062160 con 0x7fd99006da60 2024-02-10T05:57:52.739 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled grafana update... 2024-02-10T05:57:52.741 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 msgr2=0x7fd99006ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 0x7fd99006ff20 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd994000c00 tx=0x7fd99400bfb0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 msgr2=0x7fd9a4197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00b530 tx=0x7fd98c00b840 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 shutdown_connections 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9a4101fc0 0x7fd9a419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fd99006da60 0x7fd99006ff20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9a4100420 0x7fd9a4197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 --2- 172.21.15.146:0/701417482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9a4101620 0x7fd9a4197860 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 >> 172.21.15.146:0/701417482 conn(0x7fd9a40fb9c0 msgr2=0x7fd9a40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 shutdown_connections 2024-02-10T05:57:52.742 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:57:52.741+0000 7fd9a9f4a700 1 -- 172.21.15.146:0/701417482 wait complete. 2024-02-10T05:57:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[41707]: Deploying daemon prometheus.a on smithi146 2024-02-10T05:57:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-02-10T05:57:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[30020]: Deploying daemon prometheus.a on smithi146 2024-02-10T05:57:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-02-10T05:57:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:52 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:53.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:52 smithi146 ceph-mon[33795]: Deploying daemon prometheus.a on smithi146 2024-02-10T05:57:53.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]: dispatch 2024-02-10T05:57:53.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-02-10T05:57:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:52 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:57:53.764 DEBUG:teuthology.orchestra.run.smithi146:grafana.a> sudo journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T05:57:53.766 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T05:57:53.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:57:54.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 409 B/s wr, 16 op/s 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: Saving service grafana spec with placement smithi146=a;count:1 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T05:57:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[30020]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 409 B/s wr, 16 op/s 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: Saving service grafana spec with placement smithi146=a;count:1 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T05:57:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:53 smithi121 ceph-mon[41707]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:57:54.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 409 B/s wr, 16 op/s 2024-02-10T05:57:54.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: Saving service grafana spec with placement smithi146=a;count:1 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 5]}]': finished 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T05:57:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:53 smithi146 ceph-mon[33795]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 -- 172.21.15.121:0/820528535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 msgr2=0x7f18341090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 --2- 172.21.15.121:0/820528535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f18341090c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f182c00b0a0 tx=0x7f182c00b3b0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 -- 172.21.15.121:0/820528535 shutdown_connections 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 --2- 172.21.15.121:0/820528535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f18341090c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 --2- 172.21.15.121:0/820528535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 0x7f1834101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 --2- 172.21.15.121:0/820528535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 -- 172.21.15.121:0/820528535 >> 172.21.15.121:0/820528535 conn(0x7f18340fc210 msgr2=0x7f18340fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 -- 172.21.15.121:0/820528535 shutdown_connections 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.847+0000 7f183a7d9700 1 -- 172.21.15.121:0/820528535 wait complete. 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 Processor -- start 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 -- start start 2024-02-10T05:57:55.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 0x7f18341961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183419dd40 con 0x7f1834101e70 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183419de80 con 0x7f1834100c70 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f183a7d9700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183419e010 con 0x7f1834102810 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f18337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f1838d76700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f1838d76700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:53190/0 (socket says 172.21.15.121:53190) 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f1838d76700 1 -- 172.21.15.121:0/3227325147 learned_addr learned my addr 172.21.15.121:0/3227325147 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.848+0000 7f1833fff700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 0x7f18341961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 -- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 msgr2=0x7f183419d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:55.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 -- 172.21.15.121:0/3227325147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 msgr2=0x7f18341961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 0x7f18341961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 -- 172.21.15.121:0/3227325147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828009520 con 0x7f1834101e70 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f1838d76700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18337fe700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341966f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f182800ea20 tx=0x7f182800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828004220 con 0x7f1834101e70 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182c009d00 con 0x7f1834101e70 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f182800eec0 con 0x7f1834101e70 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1828003b80 con 0x7f1834101e70 2024-02-10T05:57:55.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.849+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1834069df0 con 0x7f1834101e70 2024-02-10T05:57:55.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.850+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1834050230 con 0x7f1834101e70 2024-02-10T05:57:55.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.850+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7f1828003ce0 con 0x7f1834101e70 2024-02-10T05:57:55.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.851+0000 7f18317fa700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 0x7f181c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:57:55.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.851+0000 7f1833fff700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 0x7f181c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:57:55.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.851+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f1828014070 con 0x7f1834101e70 2024-02-10T05:57:55.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.851+0000 7f1833fff700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 0x7f181c06ffd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1834101cd0 tx=0x7f1824009580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:57:55.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:55.856+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1828056e50 con 0x7f1834101e70 2024-02-10T05:57:56.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.130+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f183410bbf0 con 0x7f1834101e70 2024-02-10T05:57:56.131 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:55 smithi121 ceph-mon[41707]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 1 op/s 2024-02-10T05:57:56.131 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:55 smithi121 ceph-mon[30020]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 1 op/s 2024-02-10T05:57:56.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.134+0000 7f18317fa700 1 -- 172.21.15.121:0/3227325147 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f182805a510 con 0x7f1834101e70 2024-02-10T05:57:56.134 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-02-10T05:57:56.135 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQBkEMdlb37WBxAAsqnFjVvWUa+ipEkzbLYUBQ== 2024-02-10T05:57:56.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 msgr2=0x7f181c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:56.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 0x7f181c06ffd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1834101cd0 tx=0x7f1824009580 comp rx=0 tx=0).stop 2024-02-10T05:57:56.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 msgr2=0x7f18341966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:57:56.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341966f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f182800ea20 tx=0x7f182800ed30 comp rx=0 tx=0).stop 2024-02-10T05:57:56.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 shutdown_connections 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f181c06db10 0x7f181c06ffd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1834102810 0x7f183419d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1834100c70 0x7f18341961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 --2- 172.21.15.121:0/3227325147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1834101e70 0x7f18341966f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 >> 172.21.15.121:0/3227325147 conn(0x7f18340fc210 msgr2=0x7f18340fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:57:56.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 shutdown_connections 2024-02-10T05:57:56.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:57:56.136+0000 7f183a7d9700 1 -- 172.21.15.121:0/3227325147 wait complete. 2024-02-10T05:57:56.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:55 smithi146 ceph-mon[33795]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 127 B/s wr, 1 op/s 2024-02-10T05:57:56.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:57:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:57:56.874 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:57:56.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T05:57:56.875 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T05:57:56.917 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:57:57.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:56 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:57:57.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:57:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:57:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3227325147' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:57:58.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:57 smithi146 ceph-mon[33795]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-02-10T05:57:58.231 INFO:teuthology.orchestra.run.smithi146.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:57:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:57 smithi121 ceph-mon[41707]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-02-10T05:57:58.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:57 smithi121 ceph-mon[30020]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-02-10T05:57:59.305 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:57:59 smithi146 ceph-mon[33795]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:57:59.327 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:57:59 smithi121 ceph-mon[30020]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:57:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:57:59 smithi121 ceph-mon[41707]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 -- 172.21.15.146:0/2722134320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4102270 msgr2=0x7f0bd4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 --2- 172.21.15.146:0/2722134320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4102270 0x7f0bd4102690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009990 tx=0x7f0bc4009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 -- 172.21.15.146:0/2722134320 shutdown_connections 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 --2- 172.21.15.146:0/2722134320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4101890 0x7f0bd4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 --2- 172.21.15.146:0/2722134320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4102270 0x7f0bd4102690 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 --2- 172.21.15.146:0/2722134320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd4101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 -- 172.21.15.146:0/2722134320 >> 172.21.15.146:0/2722134320 conn(0x7f0bd40fb9c0 msgr2=0x7f0bd40fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:01.546 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 -- 172.21.15.146:0/2722134320 shutdown_connections 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.204+0000 7f0bda87a700 1 -- 172.21.15.146:0/2722134320 wait complete. 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 Processor -- start 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 -- start start 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd41008d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4102270 0x7f0bd419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd419ed70 con 0x7f0bd4100ed0 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd419eee0 con 0x7f0bd4101890 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bda87a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd419f0a0 con 0x7f0bd4102270 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:01.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bd37fe700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:36390/0 (socket says 172.21.15.146:36390) 2024-02-10T05:58:01.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.205+0000 7f0bd37fe700 1 -- 172.21.15.146:0/833493702 learned_addr learned my addr 172.21.15.146:0/833493702 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-02-10T05:58:01.548 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-02-10T05:58:01.548 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBpEMdl0vAUHRAAYKjdpc1d3q8i2D1YE2Wuhw== 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd3fff700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd41008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd8e17700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4102270 0x7f0bd419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 -- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4102270 msgr2=0x7f0bd419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4102270 0x7f0bd419e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 -- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 msgr2=0x7f0bd41008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd41008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 -- 172.21.15.146:0/833493702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40095f0 con 0x7f0bd4101890 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd3fff700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd41008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd37fe700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800b530 tx=0x7f0bc800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8011830 con 0x7f0bd4101890 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd419f380 con 0x7f0bd4101890 2024-02-10T05:58:01.549 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0bc8011e10 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8010d50 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.206+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd419f900 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.207+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd41078c0 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.208+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7f0bc8011990 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.208+0000 7f0bd17fa700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 0x7f0bbc06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.208+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f0bc808d270 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.208+0000 7f0bd3fff700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 0x7f0bbc06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.209+0000 7f0bd3fff700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 0x7f0bbc06ffd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009990 tx=0x7f0bc4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.213+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bc80567c0 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.485+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0bd410bd10 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.491+0000 7f0bd17fa700 1 -- 172.21.15.146:0/833493702 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0bc8059e80 con 0x7f0bd4101890 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 msgr2=0x7f0bbc06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:01.550 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 0x7f0bbc06ffd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009990 tx=0x7f0bc4005fb0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 msgr2=0x7f0bd419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800b530 tx=0x7f0bc800b8f0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 shutdown_connections 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f0bbc06db10 0x7f0bbc06ffd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bd4102270 0x7f0bd419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bd4100ed0 0x7f0bd41008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 --2- 172.21.15.146:0/833493702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bd4101890 0x7f0bd419c000 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 >> 172.21.15.146:0/833493702 conn(0x7f0bd40fb9c0 msgr2=0x7f0bd40fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 shutdown_connections 2024-02-10T05:58:01.551 INFO:teuthology.orchestra.run.smithi146.stderr:2024-02-10T05:58:01.493+0000 7f0bda87a700 1 -- 172.21.15.146:0/833493702 wait complete. 2024-02-10T05:58:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[30020]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:58:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.146:0/833493702' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:58:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[41707]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:58:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.146:0/833493702' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:58:02.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:01 smithi146 ceph-mon[33795]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T05:58:02.133 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.146:0/833493702' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:58:02.134 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:58:02.161 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-02-10T05:58:02.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T05:58:02.161 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T05:58:02.205 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T05:58:02.205 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T05:58:02.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T05:58:02.453 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:03.889 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:03 smithi121 ceph-mon[30020]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-02-10T05:58:03.889 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:03 smithi121 ceph-mon[41707]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 -- 172.21.15.121:0/1790767500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2954106c60 msgr2=0x7f2954107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 --2- 172.21.15.121:0/1790767500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2954106c60 0x7f2954107040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2948009990 tx=0x7f2948009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 -- 172.21.15.121:0/1790767500 shutdown_connections 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 --2- 172.21.15.121:0/1790767500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29541015c0 0x7f295410daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 --2- 172.21.15.121:0/1790767500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2954106c60 0x7f2954107040 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 --2- 172.21.15.121:0/1790767500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954100c20 0x7f2954101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 -- 172.21.15.121:0/1790767500 >> 172.21.15.121:0/1790767500 conn(0x7f2954077b50 msgr2=0x7f2954077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:04.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 -- 172.21.15.121:0/1790767500 shutdown_connections 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.132+0000 7f295af2b700 1 -- 172.21.15.121:0/1790767500 wait complete. 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 Processor -- start 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 -- start start 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2954100c20 0x7f29541965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 0x7f295419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295419b490 con 0x7f2954106c60 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295419b600 con 0x7f2954100c20 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f295af2b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295419b7c0 con 0x7f29541015c0 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f29594c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 0x7f295419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:60798/0 (socket says 172.21.15.121:60798) 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f29594c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 0x7f295419ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46450/0 (socket says 172.21.15.121:46450) 2024-02-10T05:58:04.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 -- 172.21.15.121:0/601858566 learned_addr learned my addr 172.21.15.121:0/601858566 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 -- 172.21.15.121:0/601858566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2954100c20 msgr2=0x7f29541965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2954100c20 0x7f29541965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 -- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 msgr2=0x7f295419ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 0x7f295419ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 -- 172.21.15.121:0/601858566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940009520 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.133+0000 7f2953fff700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f294000eb30 tx=0x7f294000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.134+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940009880 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.134+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f29400044a0 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.134+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29480095f0 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.134+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940003d10 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.134+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29541a3130 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.135+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2954109160 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.135+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7f2940003e70 con 0x7f29541015c0 2024-02-10T05:58:04.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.136+0000 7f2951ffb700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 0x7f294406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:04.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.136+0000 7f2958cc7700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 0x7f294406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:04.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.136+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f2940014070 con 0x7f29541015c0 2024-02-10T05:58:04.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.136+0000 7f2958cc7700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 0x7f294406ff20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f29480098e0 tx=0x7f2948005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:04.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.141+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2940056fd0 con 0x7f29541015c0 2024-02-10T05:58:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:03 smithi146 ceph-mon[33795]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 22 B/s, 1 objects/s recovering 2024-02-10T05:58:04.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.391+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f2954197910 con 0x7f29541015c0 2024-02-10T05:58:04.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.395+0000 7f2951ffb700 1 -- 172.21.15.121:0/601858566 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+174869 (secure 0 0 0) 0x7f294005a690 con 0x7f29541015c0 2024-02-10T05:58:04.399 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 msgr2=0x7f294406ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 0x7f294406ff20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f29480098e0 tx=0x7f2948005fb0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 msgr2=0x7f2954196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f294000eb30 tx=0x7f294000ee40 comp rx=0 tx=0).stop 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 shutdown_connections 2024-02-10T05:58:04.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2954106c60 0x7f295419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2954100c20 0x7f29541965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7f294406da60 0x7f294406ff20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 --2- 172.21.15.121:0/601858566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29541015c0 0x7f2954196af0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 >> 172.21.15.121:0/601858566 conn(0x7f2954077b50 msgr2=0x7f29540ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 shutdown_connections 2024-02-10T05:58:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:04.397+0000 7f295af2b700 1 -- 172.21.15.121:0/601858566 wait complete. 2024-02-10T05:58:04.986 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/601858566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:58:04.986 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/601858566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:58:05.012 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2329816948},{"type":"v1","addr":"172.21.15.121:6801","nonce":2329816948}]},"active_addr":"172.21.15.121:6801/2329816948","active_change":"2024-02-10T05:52:52.671852+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.121:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":2768398858}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":2864677123}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":1758625334}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":4218594763}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":2420027051}]}]} 2024-02-10T05:58:05.016 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T05:58:05.017 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T05:58:05.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:58:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/601858566' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:58:05.258 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:05 smithi121 ceph-mon[41707]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T05:58:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:05 smithi121 ceph-mon[30020]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T05:58:06.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:05 smithi146 ceph-mon[33795]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-02-10T05:58:06.136 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:06.136 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:05 smithi146 systemd[1]: Starting Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:06.137 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 podman[65167]: 2024-02-10 05:58:06.039506425 +0000 UTC m=+0.018883097 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-10T05:58:06.417 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 podman[65167]: 2024-02-10 05:58:06.179027399 +0000 UTC m=+0.158404068 container create c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:06.672 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 podman[65167]: 2024-02-10 05:58:06.417477517 +0000 UTC m=+0.396854202 container init c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:06.672 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 podman[65167]: 2024-02-10 05:58:06.424963385 +0000 UTC m=+0.404340057 container start c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:06.672 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.448Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.448Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.448Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi146 (none))" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.448Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.448Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.449Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.450Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.451Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.451Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.453Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.453Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.603µs 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.453Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.454Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.454Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.284µs wal_replay_duration=398.083µs wbl_replay_duration=135ns total_replay_duration=434.092µs 2024-02-10T05:58:06.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.454Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.454Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.454Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.473Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.171486ms db_storage=1.223µs remote_storage=1.05µs web_handler=328ns query_engine=588ns scrape=466.208µs scrape_sd=64.369µs notify=862ns notify_sd=1.963µs rules=17.454603ms tracing=5.357µs 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.473Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:58:06.473Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 bash[65167]: c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 2024-02-10T05:58:06.674 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:58:06 smithi146 systemd[1]: Started Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:06 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:58:07.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 -- 172.21.15.121:0/1453492106 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 msgr2=0x7fa890107040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:07.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 --2- 172.21.15.121:0/1453492106 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890107040 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa88c009990 tx=0x7fa88c009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 -- 172.21.15.121:0/1453492106 shutdown_connections 2024-02-10T05:58:07.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 --2- 172.21.15.121:0/1453492106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8901015c0 0x7fa89010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 --2- 172.21.15.121:0/1453492106 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890107040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 --2- 172.21.15.121:0/1453492106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa890100c20 0x7fa890101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 -- 172.21.15.121:0/1453492106 >> 172.21.15.121:0/1453492106 conn(0x7fa890077b50 msgr2=0x7fa890077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 -- 172.21.15.121:0/1453492106 shutdown_connections 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.148+0000 7fa898508700 1 -- 172.21.15.121:0/1453492106 wait complete. 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 Processor -- start 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 -- start start 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 0x7fa89019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:07.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa890199ec0 con 0x7fa890100c20 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa89019a030 con 0x7fa8901015c0 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa898508700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa89019a1f0 con 0x7fa890106c60 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa8962a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa896aa5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa8962a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46482/0 (socket says 172.21.15.121:46482) 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.149+0000 7fa8962a4700 1 -- 172.21.15.121:0/1086657157 learned_addr learned my addr 172.21.15.121:0/1086657157 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa895aa3700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 0x7fa89019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 -- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 msgr2=0x7fa890199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 -- 172.21.15.121:0/1086657157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 msgr2=0x7fa89019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 0x7fa89019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 -- 172.21.15.121:0/1086657157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888009c40 con 0x7fa890100c20 2024-02-10T05:58:07.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa896aa5700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa895aa3700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 0x7fa89019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8962a4700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa88c004730 tx=0x7fa88c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c01d070 con 0x7fa890100c20 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa88c005a00 con 0x7fa890100c20 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c005290 con 0x7fa890100c20 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c0095f0 con 0x7fa890100c20 2024-02-10T05:58:07.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.150+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa89019a6d0 con 0x7fa890100c20 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.151+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa890109180 con 0x7fa890100c20 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.152+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 91458+0+0 (secure 0 0 0) 0x7fa88c0053f0 con 0x7fa890100c20 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.152+0000 7fa8877fe700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 0x7fa87c06ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.152+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7207+0+0 (secure 0 0 0) 0x7fa88c08e640 con 0x7fa890100c20 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.152+0000 7fa895aa3700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 0x7fa87c06ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:07.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.153+0000 7fa895aa3700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 0x7fa87c06ff00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa890105d50 tx=0x7fa880008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:07.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.157+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa88c057b00 con 0x7fa890100c20 2024-02-10T05:58:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' 2024-02-10T05:58:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:06 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:58:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.349+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa89019a6d0 con 0x7fa890100c20 2024-02-10T05:58:07.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.350+0000 7fa8877fe700 1 -- 172.21.15.121:0/1086657157 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+22763 (secure 0 0 0) 0x7fa89019a6d0 con 0x7fa890100c20 2024-02-10T05:58:07.352 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:58:07.352 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":73,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:51:53.237116+0000","modified":"2024-02-10T05:57:53.815840+0000","last_up_change":"2024-02-10T05:57:19.157828+0000","last_in_change":"2024-02-10T05:57:00.517452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:55:13.005730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T05:57:30.772056+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T05:57:31.248915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:57:33.438445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:57:35.120371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:57:37.193482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e9f036-b22d-4b48-a265-3413e9eda4f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6803","nonce":889004683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6805","nonce":889004683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6809","nonce":889004683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6807","nonce":889004683}]},"public_addr":"172.21.15.121:6803/889004683","cluster_addr":"172.21.15.121:6805/889004683","heartbeat_back_addr":"172.21.15.121:6809/889004683","heartbeat_front_addr":"172.21.15.121:6807/889004683","state":["exists","up"]},{"osd":1,"uuid":"f733eda4-d0b0-4b09-84f2-5ec618b890db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6811","nonce":2658473683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6813","nonce":2658473683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6817","nonce":2658473683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6815","nonce":2658473683}]},"public_addr":"172.21.15.121:6811/2658473683","cluster_addr":"172.21.15.121:6813/2658473683","heartbeat_back_addr":"172.21.15.121:6817/2658473683","heartbeat_front_addr":"172.21.15.121:6815/2658473683","state":["exists","up"]},{"osd":2,"uuid":"c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6819","nonce":3662379322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6821","nonce":3662379322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6825","nonce":3662379322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6823","nonce":3662379322}]},"public_addr":"172.21.15.121:6819/3662379322","cluster_addr":"172.21.15.121:6821/3662379322","heartbeat_back_addr":"172.21.15.121:6825/3662379322","heartbeat_front_addr":"172.21.15.121:6823/3662379322","state":["exists","up"]},{"osd":3,"uuid":"b613b780-0fb2-4edf-b40f-2bc3c5782e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6827","nonce":255355367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6829","nonce":255355367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6833","nonce":255355367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6831","nonce":255355367}]},"public_addr":"172.21.15.121:6827/255355367","cluster_addr":"172.21.15.121:6829/255355367","heartbeat_back_addr":"172.21.15.121:6833/255355367","heartbeat_front_addr":"172.21.15.121:6831/255355367","state":["exists","up"]},{"osd":4,"uuid":"3bf95a4e-2fe0-4799-8e53-e730e7f49b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6801","nonce":855619125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6803","nonce":855619125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6807","nonce":855619125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6805","nonce":855619125}]},"public_addr":"172.21.15.146:6801/855619125","cluster_addr":"172.21.15.146:6803/855619125","heartbeat_back_addr":"172.21.15.146:6807/855619125","heartbeat_front_addr":"172.21.15.146:6805/855619125","state":["exists","up"]},{"osd":5,"uuid":"55de8feb-ec6c-4512-b30a-2275e97e8509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6809","nonce":3638337227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6811","nonce":3638337227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6815","nonce":3638337227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6813","nonce":3638337227}]},"public_addr":"172.21.15.146:6809/3638337227","cluster_addr":"172.21.15.146:6811/3638337227","heartbeat_back_addr":"172.21.15.146:6815/3638337227","heartbeat_front_addr":"172.21.15.146:6813/3638337227","state":["exists","up"]},{"osd":6,"uuid":"c45f670b-7a69-4f2a-98ae-ce1fba25a540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3393081156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3393081156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6823","nonce":3393081156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6821","nonce":3393081156}]},"public_addr":"172.21.15.146:6817/3393081156","cluster_addr":"172.21.15.146:6819/3393081156","heartbeat_back_addr":"172.21.15.146:6823/3393081156","heartbeat_front_addr":"172.21.15.146:6821/3393081156","state":["exists","up"]},{"osd":7,"uuid":"27b049ad-0318-4757-bb5b-54c3835d24a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6825","nonce":141569442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6827","nonce":141569442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6831","nonce":141569442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6829","nonce":141569442}]},"public_addr":"172.21.15.146:6825/141569442","cluster_addr":"172.21.15.146:6827/141569442","heartbeat_back_addr":"172.21.15.146:6831/141569442","heartbeat_front_addr":"172.21.15.146:6829/141569442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:18.533343+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:43.864363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:09.070052+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:34.771818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:58.601902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:24.563540+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.889604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:16.984463+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/92547055":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6801/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:0/3053598596":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2864864625":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2001688836":"2024-02-11T05:52:25.703724+0000","172.21.15.121:6801/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/3485345402":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/1796687448":"2024-02-11T05:52:52.671742+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 msgr2=0x7fa87c06ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 0x7fa87c06ff00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa890105d50 tx=0x7fa880008040 comp rx=0 tx=0).stop 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 msgr2=0x7fa89019f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa88c004730 tx=0x7fa88c005e80 comp rx=0 tx=0).stop 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 shutdown_connections 2024-02-10T05:58:07.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fa87c06da40 0x7fa87c06ff00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa890106c60 0x7fa890199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa890100c20 0x7fa89019f690 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 --2- 172.21.15.121:0/1086657157 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8901015c0 0x7fa89019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 >> 172.21.15.121:0/1086657157 conn(0x7fa890077b50 msgr2=0x7fa8900ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 shutdown_connections 2024-02-10T05:58:07.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:07.353+0000 7fa898508700 1 -- 172.21.15.121:0/1086657157 wait complete. 2024-02-10T05:58:07.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mon[33795]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1086657157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mon[33795]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mon[33795]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: ignoring --setuser ceph since I am not root 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: ignoring --setgroup ceph since I am not root 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mgr[35892]: -- 172.21.15.146:0/886551919 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55e95e08b080 con 0x55e95e054800 2024-02-10T05:58:07.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:07 smithi146 ceph-mgr[35892]: -- 172.21.15.146:0/886551919 <== mon.1 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e95e07e1a0 con 0x55e95e054800 2024-02-10T05:58:08.061 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[41707]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T05:58:08.061 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1086657157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:08.061 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[41707]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:58:08.061 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[41707]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setuser ceph since I am not root 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ignoring --setgroup ceph since I am not root 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[30020]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18 B/s, 1 objects/s recovering 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1086657157' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[30020]: from='mgr.14150 172.21.15.121:0/2890516042' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:58:08.062 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:07 smithi121 ceph-mon[30020]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:58:08.090 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T05:58:08.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:58:08.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:08 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:08.186+0000 7f35fccdb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:58:08.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:08 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:08.284+0000 7f35fccdb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:58:08.335 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:08.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:08.178+0000 7f1ca2356200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:58:08.672 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:08.272+0000 7f1ca2356200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:58:09.036 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:08 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:08.945+0000 7f35fccdb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:58:09.037 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.036+0000 7f35fccdb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:58:09.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:08.910+0000 7f1ca2356200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:58:09.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.004+0000 7f1ca2356200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:58:09.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.146+0000 7f1ca2356200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:58:09.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.181+0000 7f35fccdb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:58:09.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.276+0000 7f35fccdb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:58:09.591 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.242+0000 7f1ca2356200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:58:09.591 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.326+0000 7f1ca2356200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:58:09.592 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.590+0000 7f1ca2356200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:58:09.623 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.358+0000 7f35fccdb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:58:09.623 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.622+0000 7f35fccdb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:58:09.882 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.735+0000 7f35fccdb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:58:09.882 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.822+0000 7f35fccdb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:58:09.883 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.701+0000 7f1ca2356200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:58:09.883 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.787+0000 7f1ca2356200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:58:10.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 -- 172.21.15.121:0/1034433136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15406a2f0 msgr2=0x7fb15410e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 --2- 172.21.15.121:0/1034433136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15406a2f0 0x7fb15410e290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb1440098f0 tx=0x7fb144009c00 comp rx=0 tx=0).stop 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 -- 172.21.15.121:0/1034433136 shutdown_connections 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 --2- 172.21.15.121:0/1034433136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15410e7d0 0x7fb154110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 --2- 172.21.15.121:0/1034433136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb154069db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 --2- 172.21.15.121:0/1034433136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15406a2f0 0x7fb15410e290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 -- 172.21.15.121:0/1034433136 >> 172.21.15.121:0/1034433136 conn(0x7fb154077b50 msgr2=0x7fb154077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 -- 172.21.15.121:0/1034433136 shutdown_connections 2024-02-10T05:58:10.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.110+0000 7fb15bafa700 1 -- 172.21.15.121:0/1034433136 wait complete. 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 Processor -- start 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 -- start start 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15406a2f0 0x7fb15419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15410e7d0 0x7fb1541998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb154199f10 con 0x7fb1540699d0 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15419a050 con 0x7fb15406a2f0 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15bafa700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15419a210 con 0x7fb15410e7d0 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159896700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159896700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59684/0 (socket says 172.21.15.121:59684) 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159896700 1 -- 172.21.15.121:0/415816057 learned_addr learned my addr 172.21.15.121:0/415816057 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:10.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159095700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15406a2f0 0x7fb15419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb15a097700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15410e7d0 0x7fb1541998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159896700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15410e7d0 msgr2=0x7fb1541998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.111+0000 7fb159896700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15410e7d0 0x7fb1541998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb159896700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15406a2f0 msgr2=0x7fb15419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb159896700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15406a2f0 0x7fb15419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb159896700 1 -- 172.21.15.121:0/415816057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1500095f0 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb159896700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb15000ec10 tx=0x7fb15000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15000cc90 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb15000cdf0 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb144009520 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150003df0 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.112+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1541a65c0 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.113+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1541099f0 con 0x7fb1540699d0 2024-02-10T05:58:10.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.113+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 91472+0+0 (secure 0 0 0) 0x7fb150004000 con 0x7fb1540699d0 2024-02-10T05:58:10.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.114+0000 7fb14affd700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 0x7fb14006ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:10.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.114+0000 7fb159095700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 msgr2=0x7fb14006ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2329816948 2024-02-10T05:58:10.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.114+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7207+0+0 (secure 0 0 0) 0x7fb150014070 con 0x7fb1540699d0 2024-02-10T05:58:10.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.114+0000 7fb159095700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 0x7fb14006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-10T05:58:10.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.118+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb150057010 con 0x7fb1540699d0 2024-02-10T05:58:10.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:09.883+0000 7f1ca2356200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:58:10.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:10.058+0000 7f1ca2356200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:58:10.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.312+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb15419ade0 con 0x7fb1540699d0 2024-02-10T05:58:10.313 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:09 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:09.927+0000 7f35fccdb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:58:10.313 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:10 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:10.106+0000 7f35fccdb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:58:10.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.314+0000 7fb14affd700 1 -- 172.21.15.121:0/415816057 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+22763 (secure 0 0 0) 0x7fb15005a6d0 con 0x7fb1540699d0 2024-02-10T05:58:10.315 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:58:10.315 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":73,"fsid":"616c67f8-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:51:53.237116+0000","modified":"2024-02-10T05:57:53.815840+0000","last_up_change":"2024-02-10T05:57:19.157828+0000","last_in_change":"2024-02-10T05:57:00.517452+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:55:13.005730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T05:57:30.772056+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T05:57:31.248915+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:57:33.438445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:57:35.120371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:57:37.193482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d9e9f036-b22d-4b48-a265-3413e9eda4f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6803","nonce":889004683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6805","nonce":889004683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6809","nonce":889004683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":889004683},{"type":"v1","addr":"172.21.15.121:6807","nonce":889004683}]},"public_addr":"172.21.15.121:6803/889004683","cluster_addr":"172.21.15.121:6805/889004683","heartbeat_back_addr":"172.21.15.121:6809/889004683","heartbeat_front_addr":"172.21.15.121:6807/889004683","state":["exists","up"]},{"osd":1,"uuid":"f733eda4-d0b0-4b09-84f2-5ec618b890db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6811","nonce":2658473683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6813","nonce":2658473683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6817","nonce":2658473683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":2658473683},{"type":"v1","addr":"172.21.15.121:6815","nonce":2658473683}]},"public_addr":"172.21.15.121:6811/2658473683","cluster_addr":"172.21.15.121:6813/2658473683","heartbeat_back_addr":"172.21.15.121:6817/2658473683","heartbeat_front_addr":"172.21.15.121:6815/2658473683","state":["exists","up"]},{"osd":2,"uuid":"c3f3bfa8-3ce7-4040-8ce5-2c3b83fb3104","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6819","nonce":3662379322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6821","nonce":3662379322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6825","nonce":3662379322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3662379322},{"type":"v1","addr":"172.21.15.121:6823","nonce":3662379322}]},"public_addr":"172.21.15.121:6819/3662379322","cluster_addr":"172.21.15.121:6821/3662379322","heartbeat_back_addr":"172.21.15.121:6825/3662379322","heartbeat_front_addr":"172.21.15.121:6823/3662379322","state":["exists","up"]},{"osd":3,"uuid":"b613b780-0fb2-4edf-b40f-2bc3c5782e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6827","nonce":255355367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6829","nonce":255355367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6833","nonce":255355367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":255355367},{"type":"v1","addr":"172.21.15.121:6831","nonce":255355367}]},"public_addr":"172.21.15.121:6827/255355367","cluster_addr":"172.21.15.121:6829/255355367","heartbeat_back_addr":"172.21.15.121:6833/255355367","heartbeat_front_addr":"172.21.15.121:6831/255355367","state":["exists","up"]},{"osd":4,"uuid":"3bf95a4e-2fe0-4799-8e53-e730e7f49b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6801","nonce":855619125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6803","nonce":855619125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6807","nonce":855619125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":855619125},{"type":"v1","addr":"172.21.15.146:6805","nonce":855619125}]},"public_addr":"172.21.15.146:6801/855619125","cluster_addr":"172.21.15.146:6803/855619125","heartbeat_back_addr":"172.21.15.146:6807/855619125","heartbeat_front_addr":"172.21.15.146:6805/855619125","state":["exists","up"]},{"osd":5,"uuid":"55de8feb-ec6c-4512-b30a-2275e97e8509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6809","nonce":3638337227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6811","nonce":3638337227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6815","nonce":3638337227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":3638337227},{"type":"v1","addr":"172.21.15.146:6813","nonce":3638337227}]},"public_addr":"172.21.15.146:6809/3638337227","cluster_addr":"172.21.15.146:6811/3638337227","heartbeat_back_addr":"172.21.15.146:6815/3638337227","heartbeat_front_addr":"172.21.15.146:6813/3638337227","state":["exists","up"]},{"osd":6,"uuid":"c45f670b-7a69-4f2a-98ae-ce1fba25a540","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6817","nonce":3393081156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6819","nonce":3393081156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6823","nonce":3393081156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3393081156},{"type":"v1","addr":"172.21.15.146:6821","nonce":3393081156}]},"public_addr":"172.21.15.146:6817/3393081156","cluster_addr":"172.21.15.146:6819/3393081156","heartbeat_back_addr":"172.21.15.146:6823/3393081156","heartbeat_front_addr":"172.21.15.146:6821/3393081156","state":["exists","up"]},{"osd":7,"uuid":"27b049ad-0318-4757-bb5b-54c3835d24a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6825","nonce":141569442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6827","nonce":141569442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6831","nonce":141569442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":141569442},{"type":"v1","addr":"172.21.15.146:6829","nonce":141569442}]},"public_addr":"172.21.15.146:6825/141569442","cluster_addr":"172.21.15.146:6827/141569442","heartbeat_back_addr":"172.21.15.146:6831/141569442","heartbeat_front_addr":"172.21.15.146:6829/141569442","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:18.533343+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:43.864363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:09.070052+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:34.771818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:55:58.601902+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:24.563540+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:56:50.889604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:57:16.984463+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/92547055":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6801/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2724527041":"2024-02-11T05:52:52.671742+0000","172.21.15.121:0/3053598596":"2024-02-11T05:52:52.671742+0000","172.21.15.121:6800/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2864864625":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/2001688836":"2024-02-11T05:52:25.703724+0000","172.21.15.121:6801/2998711588":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/3485345402":"2024-02-11T05:52:25.703724+0000","172.21.15.121:0/1796687448":"2024-02-11T05:52:52.671742+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:58:10.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.315+0000 7fb159095700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 msgr2=0x7fb14006ff70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/2329816948 2024-02-10T05:58:10.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.315+0000 7fb159095700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 0x7fb14006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-10T05:58:10.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 msgr2=0x7fb14006ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T05:58:10.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 0x7fb14006ff70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 msgr2=0x7fb15419f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb15000ec10 tx=0x7fb15000ef20 comp rx=0 tx=0).stop 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 shutdown_connections 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:6800/2329816948,v1:172.21.15.121:6801/2329816948] conn(0x7fb14006dab0 0x7fb14006ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb15410e7d0 0x7fb1541998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1540699d0 0x7fb15419f6f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 --2- 172.21.15.121:0/415816057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb15406a2f0 0x7fb15419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 >> 172.21.15.121:0/415816057 conn(0x7fb154077b50 msgr2=0x7fb15410ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 shutdown_connections 2024-02-10T05:58:10.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:10.317+0000 7fb15bafa700 1 -- 172.21.15.121:0/415816057 wait complete. 2024-02-10T05:58:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/415816057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:10.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/415816057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:10.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/415816057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:58:11.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T05:58:11.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T05:58:11.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T05:58:11.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T05:58:11.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T05:58:11.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T05:58:11.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T05:58:11.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T05:58:11.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:11 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:11.115+0000 7f35fccdb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:58:11.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.025+0000 7f1ca2356200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:58:11.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.155+0000 7f1ca2356200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:58:11.537 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.675 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:11 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:11.361+0000 7f35fccdb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:58:11.774 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.489+0000 7f1ca2356200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:58:11.774 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.580+0000 7f1ca2356200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:58:11.812 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.814 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.815 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:11.953 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:11 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:11.788+0000 7f35fccdb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:58:11.954 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:11 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:11.904+0000 7f35fccdb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:58:12.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.774+0000 7f1ca2356200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:58:12.171 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:11.960+0000 7f1ca2356200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:58:12.259 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:12.302 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:12 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:12.107+0000 7f35fccdb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:58:12.507 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:12.245+0000 7f1ca2356200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:58:12.577 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:12 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:12.301+0000 7f35fccdb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:58:12.862 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:12 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:12.597+0000 7f35fccdb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:58:12.921 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:12.506+0000 7f1ca2356200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:58:12.921 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:12.742+0000 7f1ca2356200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:58:13.281 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:12 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:12.865+0000 7f35fccdb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:58:13.281 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:13 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:13.110+0000 7f35fccdb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:58:13.923 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:13.546+0000 7f1ca2356200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:58:14.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:13 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:13.943+0000 7f35fccdb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:58:14.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:14.119+0000 7f1ca2356200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:58:14.421 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:14.218+0000 7f1ca2356200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: Standby manager daemon x restarted 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: Standby manager daemon x started 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:58:14.691 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[30020]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:14.538+0000 7f35fccdb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:14.644+0000 7f35fccdb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: Standby manager daemon x restarted 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: Standby manager daemon x started 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:58:14.692 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-mon[41707]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:58:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: Standby manager daemon x restarted 2024-02-10T05:58:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: Standby manager daemon x started 2024-02-10T05:58:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-mon[33795]: from='mgr.? 172.21.15.146:0/1905697086' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:14.432+0000 7f1ca2356200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: [10/Feb/2024:05:58:14] ENGINE Bus STARTING 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: CherryPy Checker: 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: The Application mounted at '' has an empty config. 2024-02-10T05:58:14.922 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: 2024-02-10T05:58:14.923 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: [10/Feb/2024:05:58:14] ENGINE Serving on http://:::9283 2024-02-10T05:58:14.923 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 05:58:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x[35869]: [10/Feb/2024:05:58:14] ENGINE Bus STARTED 2024-02-10T05:58:14.994 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:14.876+0000 7f35fccdb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:58:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:14 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:14] ENGINE Bus STARTING 2024-02-10T05:58:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: CherryPy Checker: 2024-02-10T05:58:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: The Application mounted at '' has an empty config. 2024-02-10T05:58:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T05:58:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:15] ENGINE Serving on http://:::9283 2024-02-10T05:58:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:15] ENGINE Bus STARTED 2024-02-10T05:58:15.711 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T05:58:15.711 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: Active manager daemon y restarted 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: Activating manager daemon y 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: mgrmap e19: y(active, starting, since 0.0125249s), standbys: x 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:58:15.713 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:58:15.714 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: Manager daemon y is now available 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:15.715 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: Active manager daemon y restarted 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: Activating manager daemon y 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: mgrmap e19: y(active, starting, since 0.0125249s), standbys: x 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:58:15.716 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: Manager daemon y is now available 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:15.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.718 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.718 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:15.718 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: Active manager daemon y restarted 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: Activating manager daemon y 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: mgrmap e19: y(active, starting, since 0.0125249s), standbys: x 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:58:15.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:58:15.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: Manager daemon y is now available 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:15.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:58:16.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:16.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:16 smithi146 ceph-mon[33795]: [10/Feb/2024:05:58:15] ENGINE Bus STARTING 2024-02-10T05:58:16.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:16 smithi146 ceph-mon[33795]: [10/Feb/2024:05:58:15] ENGINE Serving on https://172.21.15.121:7150 2024-02-10T05:58:16.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:16 smithi146 ceph-mon[33795]: [10/Feb/2024:05:58:15] ENGINE Serving on http://172.21.15.121:8765 2024-02-10T05:58:16.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:16 smithi146 ceph-mon[33795]: [10/Feb/2024:05:58:15] ENGINE Bus STARTED 2024-02-10T05:58:16.603 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:16 smithi146 ceph-mon[33795]: mgrmap e20: y(active, since 1.01353s), standbys: x 2024-02-10T05:58:16.766 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[41707]: [10/Feb/2024:05:58:15] ENGINE Bus STARTING 2024-02-10T05:58:16.766 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[41707]: [10/Feb/2024:05:58:15] ENGINE Serving on https://172.21.15.121:7150 2024-02-10T05:58:16.766 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[41707]: [10/Feb/2024:05:58:15] ENGINE Serving on http://172.21.15.121:8765 2024-02-10T05:58:16.766 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[41707]: [10/Feb/2024:05:58:15] ENGINE Bus STARTED 2024-02-10T05:58:16.766 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[41707]: mgrmap e20: y(active, since 1.01353s), standbys: x 2024-02-10T05:58:16.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[30020]: [10/Feb/2024:05:58:15] ENGINE Bus STARTING 2024-02-10T05:58:16.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[30020]: [10/Feb/2024:05:58:15] ENGINE Serving on https://172.21.15.121:7150 2024-02-10T05:58:16.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[30020]: [10/Feb/2024:05:58:15] ENGINE Serving on http://172.21.15.121:8765 2024-02-10T05:58:16.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[30020]: [10/Feb/2024:05:58:15] ENGINE Bus STARTED 2024-02-10T05:58:16.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:16 smithi121 ceph-mon[30020]: mgrmap e20: y(active, since 1.01353s), standbys: x 2024-02-10T05:58:17.755 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:17.755 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:17 smithi121 ceph-mon[41707]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:17.755 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:17.755 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:17 smithi121 ceph-mon[30020]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:17.869 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:17.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:17 smithi146 ceph-mon[33795]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[41707]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[41707]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[30020]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[30020]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:18.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:18 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:18 smithi146 ceph-mon[33795]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:18 smithi146 ceph-mon[33795]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-10T05:58:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:18 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:18 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 -- 172.21.15.121:0/1173310992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 msgr2=0x7f9b0006aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 --2- 172.21.15.121:0/1173310992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0006aac0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9af00098f0 tx=0x7f9af0009c00 comp rx=0 tx=0).stop 2024-02-10T05:58:20.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 -- 172.21.15.121:0/1173310992 shutdown_connections 2024-02-10T05:58:20.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 --2- 172.21.15.121:0/1173310992 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b001070a0 0x7f9b0010ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 --2- 172.21.15.121:0/1173310992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0006aac0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 --2- 172.21.15.121:0/1173310992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b0006b090 0x7f9b00106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 -- 172.21.15.121:0/1173310992 >> 172.21.15.121:0/1173310992 conn(0x7f9b00076c30 msgr2=0x7f9b00077040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 -- 172.21.15.121:0/1173310992 shutdown_connections 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.169+0000 7f9b0524f700 1 -- 172.21.15.121:0/1173310992 wait complete. 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 Processor -- start 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 -- start start 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b0006b090 0x7f9b0019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b001070a0 0x7f9b001976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00197e60 con 0x7f9b0006a6e0 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00197fd0 con 0x7f9b001070a0 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.170+0000 7f9b0524f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b00198190 con 0x7f9b0006b090 2024-02-10T05:58:20.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afe59c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b0006b090 0x7f9b0019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56278/0 (socket says 172.21.15.121:56278) 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 -- 172.21.15.121:0/3460674452 learned_addr learned my addr 172.21.15.121:0/3460674452 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9aff59e700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b001070a0 0x7f9b001976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 -- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b0006b090 msgr2=0x7f9b0019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b0006b090 0x7f9b0019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 -- 172.21.15.121:0/3460674452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b001070a0 msgr2=0x7f9b001976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b001070a0 0x7f9b001976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af8009d00 con 0x7f9b0006a6e0 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afe59c700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b0006b090 0x7f9b0019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9afed9d700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0019d5e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9af0011040 tx=0x7f9af000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:20.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af001c070 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af0009520 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9af000ba60 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af0004930 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.171+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b00100c50 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.172+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b00111560 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f9af0020070 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f9ae806dac0 0x7f9ae806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f9af008e530 con 0x7f9b0006a6e0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9afe59c700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f9ae806dac0 0x7f9ae806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] conn(0x7f9ae8077950 0x7f9ae8079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ae807a4c0 con 0x7f9ae8077950 2024-02-10T05:58:20.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9af0090bf0 con 0x7f9b0006a6e0 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9aff59e700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] conn(0x7f9ae8077950 0x7f9ae8079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.173+0000 7f9afe59c700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f9ae806dac0 0x7f9ae806ff80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9af400ffd0 tx=0x7f9af400f820 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.174+0000 7f9aff59e700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] conn(0x7f9ae8077950 0x7f9ae8079df0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.174+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== osd.2 v2:172.21.15.121:6818/3662379322 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ae807a4c0 con 0x7f9ae8077950 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.190+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 --> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9b000623b0 con 0x7f9ae8077950 2024-02-10T05:58:20.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.190+0000 7f9ae7fff700 1 -- 172.21.15.121:0/3460674452 <== osd.2 v2:172.21.15.121:6818/3662379322 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9b000623b0 con 0x7f9ae8077950 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] conn(0x7f9ae8077950 msgr2=0x7f9ae8079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6818/3662379322,v1:172.21.15.121:6819/3662379322] conn(0x7f9ae8077950 0x7f9ae8079df0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f9ae806dac0 msgr2=0x7f9ae806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f9ae806dac0 0x7f9ae806ff80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9af400ffd0 tx=0x7f9af400f820 comp rx=0 tx=0).stop 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 msgr2=0x7f9b0019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 --2- 172.21.15.121:0/3460674452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b0006a6e0 0x7f9b0019d5e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9af0011040 tx=0x7f9af000fa00 comp rx=0 tx=0).stop 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9aff59e700 1 -- 172.21.15.121:0/3460674452 reap_dead start 2024-02-10T05:58:20.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 shutdown_connections 2024-02-10T05:58:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 >> 172.21.15.121:0/3460674452 conn(0x7f9b00076c30 msgr2=0x7f9b000ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 shutdown_connections 2024-02-10T05:58:20.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:20.191+0000 7f9b0524f700 1 -- 172.21.15.121:0/3460674452 wait complete. 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.622 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 3760M 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[41707]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.623 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 3760M 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.624 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:20 smithi121 ceph-mon[30020]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T05:58:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:58:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 3760M 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:20 smithi146 ceph-mon[33795]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T05:58:21.752 INFO:teuthology.orchestra.run.smithi121.stdout:77309411368 2024-02-10T05:58:21.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T05:58:22.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 -- 172.21.15.121:0/1186061914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0101860 msgr2=0x7f21d0105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 --2- 172.21.15.121:0/1186061914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0101860 0x7f21d0105820 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009990 tx=0x7f21c0009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 -- 172.21.15.121:0/1186061914 shutdown_connections 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 --2- 172.21.15.121:0/1186061914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21d0105f00 0x7f21d010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 --2- 172.21.15.121:0/1186061914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21d0100eb0 0x7f21d0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 --2- 172.21.15.121:0/1186061914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0101860 0x7f21d0105820 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 -- 172.21.15.121:0/1186061914 >> 172.21.15.121:0/1186061914 conn(0x7f21d0076c70 msgr2=0x7f21d0077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 -- 172.21.15.121:0/1186061914 shutdown_connections 2024-02-10T05:58:22.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.076+0000 7f21d8071700 1 -- 172.21.15.121:0/1186061914 wait complete. 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 Processor -- start 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 -- start start 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 0x7f21d01986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21d0101860 0x7f21d0198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21d0105f00 0x7f21d01a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01994b0 con 0x7f21d0100eb0 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a3c90 con 0x7f21d0101860 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d8071700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d01a3dd0 con 0x7f21d0105f00 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 0x7f21d01986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 0x7f21d01986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56298/0 (socket says 172.21.15.121:56298) 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 -- 172.21.15.121:0/2346669813 learned_addr learned my addr 172.21.15.121:0/2346669813 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:22.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d660e700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21d0105f00 0x7f21d01a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d560c700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21d0101860 0x7f21d0198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 -- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21d0105f00 msgr2=0x7f21d01a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21d0105f00 0x7f21d01a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 -- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21d0101860 msgr2=0x7f21d0198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21d0101860 0x7f21d0198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.077+0000 7f21d5e0d700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00095f0 con 0x7f21d0100eb0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21d5e0d700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 0x7f21d01986d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00c950 tx=0x7f21cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc0078d0 con 0x7f21d0100eb0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d01a4080 con 0x7f21d0100eb0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f21cc00f450 con 0x7f21d0100eb0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc018660 con 0x7f21d0100eb0 2024-02-10T05:58:22.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.078+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d01a4550 con 0x7f21d0100eb0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.079+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21d01104a0 con 0x7f21d0100eb0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f21cc007a30 con 0x7f21d0100eb0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f21bc06dac0 0x7f21bc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f21cc08d6d0 con 0x7f21d0100eb0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21d560c700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f21bc06dac0 0x7f21bc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] conn(0x7f21bc077950 0x7f21bc079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21bc07a4c0 con 0x7f21bc077950 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21cc08fd90 con 0x7f21d0100eb0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.080+0000 7f21d560c700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f21bc06dac0 0x7f21bc06ff80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21c0006010 tx=0x7f21c000b260 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.081+0000 7f21d660e700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] conn(0x7f21bc077950 0x7f21bc079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.081+0000 7f21d660e700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] conn(0x7f21bc077950 0x7f21bc079df0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.082+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== osd.6 v2:172.21.15.146:6816/3393081156 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21bc07a4c0 con 0x7f21bc077950 2024-02-10T05:58:22.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.099+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 --> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21d00623b0 con 0x7f21bc077950 2024-02-10T05:58:22.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.099+0000 7f21c6ffd700 1 -- 172.21.15.121:0/2346669813 <== osd.6 v2:172.21.15.146:6816/3393081156 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f21d00623b0 con 0x7f21bc077950 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] conn(0x7f21bc077950 msgr2=0x7f21bc079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.146:6816/3393081156,v1:172.21.15.146:6817/3393081156] conn(0x7f21bc077950 0x7f21bc079df0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f21bc06dac0 msgr2=0x7f21bc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f21bc06dac0 0x7f21bc06ff80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21c0006010 tx=0x7f21c000b260 comp rx=0 tx=0).stop 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 msgr2=0x7f21d01986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 --2- 172.21.15.121:0/2346669813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21d0100eb0 0x7f21d01986d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f21cc00c950 tx=0x7f21cc00cc60 comp rx=0 tx=0).stop 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d660e700 1 -- 172.21.15.121:0/2346669813 reap_dead start 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 shutdown_connections 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 >> 172.21.15.121:0/2346669813 conn(0x7f21d0076c70 msgr2=0x7f21d00fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 shutdown_connections 2024-02-10T05:58:22.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.100+0000 7f21d8071700 1 -- 172.21.15.121:0/2346669813 wait complete. 2024-02-10T05:58:22.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/3798109425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 msgr2=0x7fd2f410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 --2- 172.21.15.121:0/3798109425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f410ec50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b0a0 tx=0x7fd2ec00b3b0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/3798109425 shutdown_connections 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 --2- 172.21.15.121:0/3798109425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f410ec50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 --2- 172.21.15.121:0/3798109425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f406a660 0x7fd2f406aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 --2- 172.21.15.121:0/3798109425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f41069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/3798109425 >> 172.21.15.121:0/3798109425 conn(0x7fd2f4076c70 msgr2=0x7fd2f4077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:22.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/3798109425 shutdown_connections 2024-02-10T05:58:22.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.284+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/3798109425 wait complete. 2024-02-10T05:58:22.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 Processor -- start 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 -- start start 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f406a660 0x7fd2f4100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f4101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f41017d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f41a5c70 con 0x7fd2f41070d0 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f41a5de0 con 0x7fd2f406a660 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.285+0000 7fd2fa2dd700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f41a5fa0 con 0x7fd2f406b010 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f37fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f4101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f37fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f4101290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38124/0 (socket says 172.21.15.121:38124) 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f37fe700 1 -- 172.21.15.121:0/1164583437 learned_addr learned my addr 172.21.15.121:0/1164583437 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:22.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f41017d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f3fff700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f406a660 0x7fd2f4100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 -- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 msgr2=0x7fd2f4101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f4101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 -- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f406a660 msgr2=0x7fd2f4100d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f406a660 0x7fd2f4100d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8009520 con 0x7fd2f41070d0 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f37fe700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2f406b010 0x7fd2f4101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:22.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.286+0000 7fd2f887a700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f41017d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b070 tx=0x7fd2ec0078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.287+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00e040 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.287+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec009d00 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.287+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd2ec01d470 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.287+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0052c0 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.287+0000 7fd2fa2dd700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f41a6550 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.288+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2d4000f80 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.288+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fd2ec004a50 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f17fa700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2dc06dac0 0x7fd2dc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fd2ec08ebb0 con 0x7fd2f41070d0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f3fff700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2dc06dac0 0x7fd2dc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f17fa700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] conn(0x7fd2dc077950 0x7fd2dc079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:22.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2dc07a4c0 con 0x7fd2dc077950 2024-02-10T05:58:22.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2ec0932c0 con 0x7fd2f41070d0 2024-02-10T05:58:22.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f3fff700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2dc06dac0 0x7fd2dc06ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400ac50 tx=0x7fd2e400a380 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.289+0000 7fd2f37fe700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] conn(0x7fd2dc077950 0x7fd2dc079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:22.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.290+0000 7fd2f37fe700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] conn(0x7fd2dc077950 0x7fd2dc079df0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:22.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.291+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== osd.5 v2:172.21.15.146:6808/3638337227 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd2dc07a4c0 con 0x7fd2dc077950 2024-02-10T05:58:22.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.308+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 --> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2d4002c60 con 0x7fd2dc077950 2024-02-10T05:58:22.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.308+0000 7fd2f17fa700 1 -- 172.21.15.121:0/1164583437 <== osd.5 v2:172.21.15.146:6808/3638337227 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd2d4002c60 con 0x7fd2dc077950 2024-02-10T05:58:22.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] conn(0x7fd2dc077950 msgr2=0x7fd2dc079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.146:6808/3638337227,v1:172.21.15.146:6809/3638337227] conn(0x7fd2dc077950 0x7fd2dc079df0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2dc06dac0 msgr2=0x7fd2dc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2dc06dac0 0x7fd2dc06ff80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400ac50 tx=0x7fd2e400a380 comp rx=0 tx=0).stop 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 msgr2=0x7fd2f41017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 --2- 172.21.15.121:0/1164583437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2f41070d0 0x7fd2f41017d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00b070 tx=0x7fd2ec0078e0 comp rx=0 tx=0).stop 2024-02-10T05:58:22.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2f887a700 1 -- 172.21.15.121:0/1164583437 reap_dead start 2024-02-10T05:58:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 shutdown_connections 2024-02-10T05:58:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 >> 172.21.15.121:0/1164583437 conn(0x7fd2f4076c70 msgr2=0x7fd2f4108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 shutdown_connections 2024-02-10T05:58:22.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:22.309+0000 7fd2daffd700 1 -- 172.21.15.121:0/1164583437 wait complete. 2024-02-10T05:58:22.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:22 smithi121 ceph-mon[41707]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:58:22.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:22 smithi121 ceph-mon[30020]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:58:22.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:22 smithi146 ceph-mon[33795]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:58:22.903 INFO:teuthology.orchestra.run.smithi121.stdout:201863462932 2024-02-10T05:58:22.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T05:58:23.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/1919318569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 msgr2=0x7fdfd8106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/1919318569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 0x7fdfd8106980 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009990 tx=0x7fdfc8009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/1919318569 shutdown_connections 2024-02-10T05:58:23.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/1919318569 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8107060 0x7fdfd810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/1919318569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd8102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/1919318569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 0x7fdfd8106980 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.092+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/1919318569 >> 172.21.15.121:0/1919318569 conn(0x7fdfd80fd7f0 msgr2=0x7fdfd80ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.093+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/1919318569 shutdown_connections 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.093+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/1919318569 wait complete. 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.093+0000 7fdfdfeb5700 1 Processor -- start 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.093+0000 7fdfdfeb5700 1 -- start start 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 0x7fdfd819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8107060 0x7fdfd8197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8197de0 con 0x7fdfd8101f80 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8197f50 con 0x7fdfd8107060 2024-02-10T05:58:23.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd8198110 con 0x7fdfd8102930 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd819d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56328/0 (socket says 172.21.15.121:56328) 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdd450700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 0x7fdfd819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 -- 172.21.15.121:0/457294773 learned_addr learned my addr 172.21.15.121:0/457294773 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfde452700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8107060 0x7fdfd8197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 -- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 msgr2=0x7fdfd819daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfd8102930 0x7fdfd819daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 -- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8107060 msgr2=0x7fdfd8197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfd8107060 0x7fdfd8197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4009520 con 0x7fdfd8101f80 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfddc51700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd819d560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4010e40 tx=0x7fdfd400b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4015070 con 0x7fdfd8101f80 2024-02-10T05:58:23.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fdfd40044a0 con 0x7fdfd8101f80 2024-02-10T05:58:23.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc80095f0 con 0x7fdfd8101f80 2024-02-10T05:58:23.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4003db0 con 0x7fdfd8101f80 2024-02-10T05:58:23.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.094+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd8073f50 con 0x7fdfd8101f80 2024-02-10T05:58:23.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.095+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdfd81115b0 con 0x7fdfd8101f80 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fdfd4004000 con 0x7fdfd8101f80 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fdfc406dac0 0x7fdfc406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fdfd40911f0 con 0x7fdfd8101f80 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfdd450700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fdfc406dac0 0x7fdfc406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] conn(0x7fdfc4077950 0x7fdfc4079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc407a4c0 con 0x7fdfc4077950 2024-02-10T05:58:23.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.096+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfd40938b0 con 0x7fdfd8101f80 2024-02-10T05:58:23.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.097+0000 7fdfde452700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] conn(0x7fdfc4077950 0x7fdfc4079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:23.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.097+0000 7fdfdd450700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fdfc406dac0 0x7fdfc406ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80058e0 tx=0x7fdfc8005960 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:23.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.097+0000 7fdfde452700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] conn(0x7fdfc4077950 0x7fdfc4079df0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:23.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.098+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== osd.4 v2:172.21.15.146:6800/855619125 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdfc407a4c0 con 0x7fdfc4077950 2024-02-10T05:58:23.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.116+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 --> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdfd80fe730 con 0x7fdfc4077950 2024-02-10T05:58:23.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.116+0000 7fdfceffd700 1 -- 172.21.15.121:0/457294773 <== osd.4 v2:172.21.15.146:6800/855619125 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdfd80fe730 con 0x7fdfc4077950 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] conn(0x7fdfc4077950 msgr2=0x7fdfc4079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.146:6800/855619125,v1:172.21.15.146:6801/855619125] conn(0x7fdfc4077950 0x7fdfc4079df0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fdfc406dac0 msgr2=0x7fdfc406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fdfc406dac0 0x7fdfc406ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80058e0 tx=0x7fdfc8005960 comp rx=0 tx=0).stop 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 msgr2=0x7fdfd819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 --2- 172.21.15.121:0/457294773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfd8101f80 0x7fdfd819d560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdfd4010e40 tx=0x7fdfd400b5a0 comp rx=0 tx=0).stop 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfde452700 1 -- 172.21.15.121:0/457294773 reap_dead start 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 shutdown_connections 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 >> 172.21.15.121:0/457294773 conn(0x7fdfd80fd7f0 msgr2=0x7fdfd80fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 shutdown_connections 2024-02-10T05:58:23.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.117+0000 7fdfdfeb5700 1 -- 172.21.15.121:0/457294773 wait complete. 2024-02-10T05:58:24.429 INFO:teuthology.orchestra.run.smithi121.stdout:133143986206 2024-02-10T05:58:24.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T05:58:24.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:24 smithi146 ceph-mon[33795]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:58:24.755 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:24 smithi121 ceph-mon[41707]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:58:24.756 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:24 smithi121 ceph-mon[30020]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:58:25.145 INFO:teuthology.orchestra.run.smithi121.stdout:167503724569 2024-02-10T05:58:25.145 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T05:58:25.263 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:58:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:58:25.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 -- 172.21.15.121:0/1002095064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11281008c0 msgr2=0x7f1128100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 --2- 172.21.15.121:0/1002095064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11281008c0 0x7f1128100ca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f11180098f0 tx=0x7f1118009c00 comp rx=0 tx=0).stop 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 -- 172.21.15.121:0/1002095064 shutdown_connections 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 --2- 172.21.15.121:0/1002095064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281070e0 0x7f112810ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 --2- 172.21.15.121:0/1002095064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11281008c0 0x7f1128100ca0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 --2- 172.21.15.121:0/1002095064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1128101270 0x7f11281069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 -- 172.21.15.121:0/1002095064 >> 172.21.15.121:0/1002095064 conn(0x7f1128075f60 msgr2=0x7f1128076370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 -- 172.21.15.121:0/1002095064 shutdown_connections 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.869+0000 7f112d205700 1 -- 172.21.15.121:0/1002095064 wait complete. 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 Processor -- start 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 -- start start 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 0x7f112819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 0x7f112819d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:25.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11281070e0 0x7f1128197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1128197ca0 con 0x7f1128101270 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1128197de0 con 0x7f11281070e0 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f112d205700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1128197fa0 con 0x7f11281008c0 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f111ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 0x7f112819d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f1126d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 0x7f112819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f111ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 0x7f112819d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56364/0 (socket says 172.21.15.121:56364) 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.870+0000 7f1126d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 0x7f112819d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38182/0 (socket says 172.21.15.121:38182) 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 -- 172.21.15.121:0/2941392483 learned_addr learned my addr 172.21.15.121:0/2941392483 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f112759e700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11281070e0 0x7f1128197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 -- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 msgr2=0x7f112819d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 0x7f112819d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 -- 172.21.15.121:0/2941392483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11281070e0 msgr2=0x7f1128197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11281070e0 0x7f1128197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1120009d00 con 0x7f1128101270 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f1126d9d700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11281008c0 0x7f112819d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f111ffff700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 0x7f112819d970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f111000c9b0 tx=0x7f111000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1110007aa0 con 0x7f1128101270 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1110007c00 con 0x7f1128101270 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1118009520 con 0x7f1128101270 2024-02-10T05:58:25.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1110018400 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.871+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11281a4300 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.872+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1128111630 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f111000f450 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f111406dac0 0x7f111406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f111008d7a0 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1126d9d700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f111406dac0 0x7f111406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] conn(0x7f1114077950 0x7f1114079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111407a4c0 con 0x7f1114077950 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f111008fe60 con 0x7f1128101270 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.873+0000 7f112759e700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] conn(0x7f1114077950 0x7f1114079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.874+0000 7f1126d9d700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f111406dac0 0x7f111406ff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11180098c0 tx=0x7f1118019040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.874+0000 7f112759e700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] conn(0x7f1114077950 0x7f1114079df0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.874+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== osd.3 v2:172.21.15.121:6826/255355367 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f111407a4c0 con 0x7f1114077950 2024-02-10T05:58:25.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 --> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11280623b0 con 0x7f1114077950 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f1124d99700 1 -- 172.21.15.121:0/2941392483 <== osd.3 v2:172.21.15.121:6826/255355367 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f11280623b0 con 0x7f1114077950 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] conn(0x7f1114077950 msgr2=0x7f1114079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6826/255355367,v1:172.21.15.121:6827/255355367] conn(0x7f1114077950 0x7f1114079df0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f111406dac0 msgr2=0x7f111406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f111406dac0 0x7f111406ff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11180098c0 tx=0x7f1118019040 comp rx=0 tx=0).stop 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 msgr2=0x7f112819d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 --2- 172.21.15.121:0/2941392483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1128101270 0x7f112819d970 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f111000c9b0 tx=0x7f111000ccc0 comp rx=0 tx=0).stop 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112759e700 1 -- 172.21.15.121:0/2941392483 reap_dead start 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 shutdown_connections 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.891+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 >> 172.21.15.121:0/2941392483 conn(0x7f1128075f60 msgr2=0x7f11280ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.892+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 shutdown_connections 2024-02-10T05:58:25.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.892+0000 7f112d205700 1 -- 172.21.15.121:0/2941392483 wait complete. 2024-02-10T05:58:26.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:25 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:26.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:26 smithi146 ceph-mon[33795]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:58:26.695 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:26 smithi121 ceph-mon[41707]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:58:26.695 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:26 smithi121 ceph-mon[30020]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:58:26.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 -- 172.21.15.121:0/104352208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 msgr2=0x7f610010eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 --2- 172.21.15.121:0/104352208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f610010eb70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f60f800b0d0 tx=0x7f60f800b3e0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 -- 172.21.15.121:0/104352208 shutdown_connections 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 --2- 172.21.15.121:0/104352208 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f610010eb70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 --2- 172.21.15.121:0/104352208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 0x7f6100108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 --2- 172.21.15.121:0/104352208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 0x7f61001021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 -- 172.21.15.121:0/104352208 >> 172.21.15.121:0/104352208 conn(0x7f6100078e50 msgr2=0x7f6100079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 -- 172.21.15.121:0/104352208 shutdown_connections 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.660+0000 7f61063c1700 1 -- 172.21.15.121:0/104352208 wait complete. 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 Processor -- start 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 -- start start 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 0x7f6100074020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f6100074560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 0x7f6100074aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001a5c70 con 0x7f6100102730 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61001a5de0 con 0x7f6100107d50 2024-02-10T05:58:26.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f61063c1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61001a5fa0 con 0x7f6100101d70 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f60ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f6100074560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f60fffff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 0x7f6100074020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f60ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f6100074560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56338/0 (socket says 172.21.15.121:56338) 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.661+0000 7f60ff7fe700 1 -- 172.21.15.121:0/810496069 learned_addr learned my addr 172.21.15.121:0/810496069 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f610495e700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 0x7f6100074aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 -- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 msgr2=0x7f6100074020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 0x7f6100074020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 -- 172.21.15.121:0/810496069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 msgr2=0x7f6100074aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 0x7f6100074aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f4009520 con 0x7f6100102730 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60fffff700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6100101d70 0x7f6100074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f610495e700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100107d50 0x7f6100074aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:58:26.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60ff7fe700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f6100074560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60f400eb30 tx=0x7f60f400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4009880 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f8009d00 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f60f40044a0 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4003d10 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.662+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61001a6640 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.663+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6100111570 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.663+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f60f4003e70 con 0x7f6100102730 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fd7fa700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f60e806dac0 0x7f60e806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fffff700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f60e806dac0 0x7f60e806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:26.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f60f4014070 con 0x7f6100102730 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fd7fa700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] conn(0x7f60e8077950 0x7f60e8079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60e807a4c0 con 0x7f60e8077950 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60f40922c0 con 0x7f6100102730 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f610495e700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] conn(0x7f60e8077950 0x7f60e8079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.664+0000 7f60fffff700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f60e806dac0 0x7f60e806ff80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009de0 tx=0x7f60f0009190 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.665+0000 7f610495e700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] conn(0x7f60e8077950 0x7f60e8079df0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.665+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== osd.1 v2:172.21.15.121:6810/2658473683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f60e807a4c0 con 0x7f60e8077950 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.681+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 --> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6100107270 con 0x7f60e8077950 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f60fd7fa700 1 -- 172.21.15.121:0/810496069 <== osd.1 v2:172.21.15.121:6810/2658473683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6100107270 con 0x7f60e8077950 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] conn(0x7f60e8077950 msgr2=0x7f60e8079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6810/2658473683,v1:172.21.15.121:6811/2658473683] conn(0x7f60e8077950 0x7f60e8079df0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:26.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f60e806dac0 msgr2=0x7f60e806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f60e806dac0 0x7f60e806ff80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009de0 tx=0x7f60f0009190 comp rx=0 tx=0).stop 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 msgr2=0x7f6100074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 --2- 172.21.15.121:0/810496069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6100102730 0x7f6100074560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60f400eb30 tx=0x7f60f400ee40 comp rx=0 tx=0).stop 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f610495e700 1 -- 172.21.15.121:0/810496069 reap_dead start 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 shutdown_connections 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 >> 172.21.15.121:0/810496069 conn(0x7f6100078e50 msgr2=0x7f6100105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 shutdown_connections 2024-02-10T05:58:26.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:23.682+0000 7f61063c1700 1 -- 172.21.15.121:0/810496069 wait complete. 2024-02-10T05:58:27.250 INFO:teuthology.orchestra.run.smithi121.stdout:111669149732 2024-02-10T05:58:27.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T05:58:27.351 INFO:teuthology.orchestra.run.smithi121.stdout:55834574893 2024-02-10T05:58:27.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T05:58:27.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 -- 172.21.15.121:0/3747766727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 msgr2=0x7fb038102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/3747766727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb038102340 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb03000b0a0 tx=0x7fb03000b3b0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 -- 172.21.15.121:0/3747766727 shutdown_connections 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/3747766727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 0x7fb03810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/3747766727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb038102340 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/3747766727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 0x7fb038106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 -- 172.21.15.121:0/3747766727 >> 172.21.15.121:0/3747766727 conn(0x7fb0380fd7d0 msgr2=0x7fb0380ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 -- 172.21.15.121:0/3747766727 shutdown_connections 2024-02-10T05:58:27.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 -- 172.21.15.121:0/3747766727 wait complete. 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.603+0000 7fb03ddd1700 1 Processor -- start 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 -- start start 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb03819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 0x7fb03819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 0x7fb0381977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038197f50 con 0x7fb038101f60 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0381980c0 con 0x7fb038102910 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb03ddd1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb038198280 con 0x7fb038107040 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb03819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb037fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 0x7fb0381977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb03819d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33652/0 (socket says 172.21.15.121:33652) 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 -- 172.21.15.121:0/2050850301 learned_addr learned my addr 172.21.15.121:0/2050850301 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb036ffd700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 0x7fb03819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 -- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 msgr2=0x7fb0381977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 0x7fb0381977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 -- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 msgr2=0x7fb03819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 0x7fb03819dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb028009520 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb037fff700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb038107040 0x7fb0381977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb036ffd700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038102910 0x7fb03819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.604+0000 7fb0377fe700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb03819d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb03000b780 tx=0x7fb030007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03000e040 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb030004c50 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb030009d00 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0300052c0 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0381a4430 con 0x7fb038101f60 2024-02-10T05:58:27.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.605+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb038111550 con 0x7fb038101f60 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.606+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fb03001d470 con 0x7fb038101f60 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.606+0000 7fb034ff9700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb02006dac0 0x7fb02006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fb03008ed40 con 0x7fb038101f60 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb036ffd700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb02006dac0 0x7fb02006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb034ff9700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] conn(0x7fb020077950 0x7fb020079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb02007a4c0 con 0x7fb020077950 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb030091400 con 0x7fb038101f60 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb036ffd700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb02006dac0 0x7fb02006ff80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb038198f10 tx=0x7fb02c005f50 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb037fff700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] conn(0x7fb020077950 0x7fb020079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.607+0000 7fb037fff700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] conn(0x7fb020077950 0x7fb020079df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.608+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== osd.7 v2:172.21.15.146:6824/141569442 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb02007a4c0 con 0x7fb020077950 2024-02-10T05:58:27.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.630+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 --> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb0380623b0 con 0x7fb020077950 2024-02-10T05:58:27.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.630+0000 7fb034ff9700 1 -- 172.21.15.121:0/2050850301 <== osd.7 v2:172.21.15.146:6824/141569442 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb0380623b0 con 0x7fb020077950 2024-02-10T05:58:27.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] conn(0x7fb020077950 msgr2=0x7fb020079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.146:6824/141569442,v1:172.21.15.146:6825/141569442] conn(0x7fb020077950 0x7fb020079df0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb02006dac0 msgr2=0x7fb02006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb02006dac0 0x7fb02006ff80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb038198f10 tx=0x7fb02c005f50 comp rx=0 tx=0).stop 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 msgr2=0x7fb03819d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 --2- 172.21.15.121:0/2050850301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038101f60 0x7fb03819d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb03000b780 tx=0x7fb030007a60 comp rx=0 tx=0).stop 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb037fff700 1 -- 172.21.15.121:0/2050850301 reap_dead start 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 shutdown_connections 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.631+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 >> 172.21.15.121:0/2050850301 conn(0x7fb0380fd7d0 msgr2=0x7fb0380feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:27.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.632+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 shutdown_connections 2024-02-10T05:58:27.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:27.632+0000 7fb03ddd1700 1 -- 172.21.15.121:0/2050850301 wait complete. 2024-02-10T05:58:27.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 -- 172.21.15.121:0/2667471012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281057b0 msgr2=0x7fc02810d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2667471012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281057b0 0x7fc02810d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc020009fc0 tx=0x7fc02000b330 comp rx=0 tx=0).stop 2024-02-10T05:58:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 -- 172.21.15.121:0/2667471012 shutdown_connections 2024-02-10T05:58:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2667471012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281057b0 0x7fc02810d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2667471012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2667471012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281012d0 0x7fc028105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 -- 172.21.15.121:0/2667471012 >> 172.21.15.121:0/2667471012 conn(0x7fc028075fa0 msgr2=0x7fc0280763b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 -- 172.21.15.121:0/2667471012 shutdown_connections 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.979+0000 7fc02c99b700 1 -- 172.21.15.121:0/2667471012 wait complete. 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 Processor -- start 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 -- start start 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281012d0 0x7fc028198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281057b0 0x7fc0281a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc028199580 con 0x7fc0281012d0 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0281a3d40 con 0x7fc0281057b0 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02c99b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0281a3e80 con 0x7fc028100920 2024-02-10T05:58:27.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc025d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281012d0 0x7fc028198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02659c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028198780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38200/0 (socket says 172.21.15.121:38200) 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc02659c700 1 -- 172.21.15.121:0/2006908104 learned_addr learned my addr 172.21.15.121:0/2006908104 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.980+0000 7fc026d9d700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281057b0 0x7fc0281a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 -- 172.21.15.121:0/2006908104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281057b0 msgr2=0x7fc0281a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281057b0 0x7fc0281a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 -- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281012d0 msgr2=0x7fc028198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281012d0 0x7fc028198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180095f0 con 0x7fc028100920 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc025d9b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0281012d0 0x7fc028198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc026d9d700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0281057b0 0x7fc0281a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02659c700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028198780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc01800ebf0 tx=0x7fc01800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018009a10 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc01800cd40 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc020009c40 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018018680 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.981+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0281a44d0 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.982+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc02810fc40 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.982+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fc018010430 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc0177fe700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fc01006dac0 0x7fc01006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc025d9b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fc01006dac0 0x7fc01006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fc018014070 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc0177fe700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] conn(0x7fc010077950 0x7fc010079df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc01007a4c0 con 0x7fc010077950 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc018090340 con 0x7fc028100920 2024-02-10T05:58:27.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc026d9d700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] conn(0x7fc010077950 0x7fc010079df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc025d9b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fc01006dac0 0x7fc01006ff80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc028076610 tx=0x7fc01c00a380 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.983+0000 7fc026d9d700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] conn(0x7fc010077950 0x7fc010079df0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:25.984+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== osd.0 v2:172.21.15.121:6802/889004683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc01007a4c0 con 0x7fc010077950 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 --> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc028104930 con 0x7fc010077950 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc0177fe700 1 -- 172.21.15.121:0/2006908104 <== osd.0 v2:172.21.15.121:6802/889004683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc028104930 con 0x7fc010077950 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] conn(0x7fc010077950 msgr2=0x7fc010079df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6802/889004683,v1:172.21.15.121:6803/889004683] conn(0x7fc010077950 0x7fc010079df0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:27.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fc01006dac0 msgr2=0x7fc01006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fc01006dac0 0x7fc01006ff80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc028076610 tx=0x7fc01c00a380 comp rx=0 tx=0).stop 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 msgr2=0x7fc028198780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 --2- 172.21.15.121:0/2006908104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc028100920 0x7fc028198780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc01800ebf0 tx=0x7fc01800ef00 comp rx=0 tx=0).stop 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc026d9d700 1 -- 172.21.15.121:0/2006908104 reap_dead start 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 shutdown_connections 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.000+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 >> 172.21.15.121:0/2006908104 conn(0x7fc028075fa0 msgr2=0x7fc028103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.001+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 shutdown_connections 2024-02-10T05:58:27.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:26.001+0000 7fc02c99b700 1 -- 172.21.15.121:0/2006908104 wait complete. 2024-02-10T05:58:28.321 INFO:teuthology.orchestra.run.smithi121.stdout:34359738420 2024-02-10T05:58:28.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T05:58:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:28 smithi121 ceph-mon[41707]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:58:28.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:28 smithi121 ceph-mon[30020]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:58:28.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:28 smithi146 ceph-mon[33795]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:58:29.177 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.180 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.182 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.652 INFO:teuthology.orchestra.run.smithi121.stdout:240518168592 2024-02-10T05:58:29.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T05:58:29.657 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.662 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:29.666 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:30.443 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:30 smithi121 ceph-mon[41707]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:30.443 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:30 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:30.443 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:30 smithi121 ceph-mon[30020]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:30.443 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:30 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:30 smithi146 ceph-mon[33795]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:30 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:32 smithi146 ceph-mon[33795]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:32.676 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:32 smithi121 ceph-mon[41707]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:32.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:32 smithi121 ceph-mon[30020]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:58:34.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:34 smithi121 ceph-mon[30020]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:34 smithi121 ceph-mon[41707]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:34 smithi146 ceph-mon[33795]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:35.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:58:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:58:35.549 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3227179742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:58:35.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3227179742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:58:35.670 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3227179742' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:58:36.381 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:35 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 -- 172.21.15.121:0/2763648596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601008c0 msgr2=0x7f9960100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 --2- 172.21.15.121:0/2763648596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601008c0 0x7f9960100ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9950009990 tx=0x7f9950009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 -- 172.21.15.121:0/2763648596 shutdown_connections 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 --2- 172.21.15.121:0/2763648596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99601070e0 0x7f996010ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 --2- 172.21.15.121:0/2763648596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601008c0 0x7f9960100ca0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 --2- 172.21.15.121:0/2763648596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9960101270 0x7f99601069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 -- 172.21.15.121:0/2763648596 >> 172.21.15.121:0/2763648596 conn(0x7f9960075f60 msgr2=0x7f9960076370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 -- 172.21.15.121:0/2763648596 shutdown_connections 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.928+0000 7f9964cf0700 1 -- 172.21.15.121:0/2763648596 wait complete. 2024-02-10T05:58:36.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 Processor -- start 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 -- start start 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 0x7f9960101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 0x7f9960101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5b90 con 0x7f99601070e0 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5d00 con 0x7f99601008c0 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f9964cf0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99601a5ec0 con 0x7f9960101270 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995dd9b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 0x7f9960101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995e59c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 0x7f9960101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33668/0 (socket says 172.21.15.121:33668) 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995e59c700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 0x7f9960101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.121:44110/0 (socket says 172.21.15.121:44110) 2024-02-10T05:58:36.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 -- 172.21.15.121:0/3227179742 learned_addr learned my addr 172.21.15.121:0/3227179742 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stdout:133143986208 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 -- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 msgr2=0x7f9960101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 0x7f9960101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 -- 172.21.15.121:0/3227179742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 msgr2=0x7f9960101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 0x7f9960101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 -- 172.21.15.121:0/3227179742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500095f0 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995dd9b700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 0x7f9960101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f995ed9d700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f995800ef30 tx=0x7f995800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.929+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958018070 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.930+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99601a6140 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.930+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f995800f040 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.930+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800cba0 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.930+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601a6640 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.930+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996010a340 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.931+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f995800cd00 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.932+0000 7f994f7fe700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 0x7f994806ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.932+0000 7f995e59c700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 0x7f994806ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.932+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f995808d3c0 con 0x7f99601070e0 2024-02-10T05:58:36.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.932+0000 7f995e59c700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 0x7f994806ff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f995000b2e0 tx=0x7f9950005040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:34.936+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f995805a300 con 0x7f99601070e0 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.137+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9960102d20 con 0x7f99601070e0 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.137+0000 7f994f7fe700 1 -- 172.21.15.121:0/3227179742 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9958059e90 con 0x7f99601070e0 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.139+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 msgr2=0x7f994806ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.139+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 0x7f994806ff80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f995000b2e0 tx=0x7f9950005040 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.139+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 msgr2=0x7f99601022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.139+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f995800ef30 tx=0x7f995800c3a0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.139+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 shutdown_connections 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f994806dac0 0x7f994806ff80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99601070e0 0x7f99601022c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601008c0 0x7f9960101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 --2- 172.21.15.121:0/3227179742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9960101270 0x7f9960101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 >> 172.21.15.121:0/3227179742 conn(0x7f9960075f60 msgr2=0x7f99600ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:36.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 shutdown_connections 2024-02-10T05:58:36.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:35.140+0000 7f9964cf0700 1 -- 172.21.15.121:0/3227179742 wait complete. 2024-02-10T05:58:36.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:36 smithi146 ceph-mon[33795]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:36.717 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:36 smithi121 ceph-mon[41707]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:36.717 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:36 smithi121 ceph-mon[30020]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:37.027 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986208 for osd.4 2024-02-10T05:58:37.027 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:38.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:38 smithi121 ceph-mon[30020]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:38.541 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:38 smithi121 ceph-mon[41707]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:38.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:38 smithi146 ceph-mon[33795]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 -- 172.21.15.121:0/1514040452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594100920 msgr2=0x7fb594100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/1514040452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594100920 0x7fb594100d00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb590009990 tx=0x7fb590009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 -- 172.21.15.121:0/1514040452 shutdown_connections 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/1514040452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5941070f0 0x7fb59410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/1514040452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594100920 0x7fb594100d00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/1514040452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941012d0 0x7fb594106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 -- 172.21.15.121:0/1514040452 >> 172.21.15.121:0/1514040452 conn(0x7fb594075fa0 msgr2=0x7fb5940763b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 -- 172.21.15.121:0/1514040452 shutdown_connections 2024-02-10T05:58:38.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.918+0000 7fb59b2b3700 1 -- 172.21.15.121:0/1514040452 wait complete. 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 Processor -- start 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 -- start start 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 0x7fb5941018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941070f0 0x7fb594102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb594103840 con 0x7fb5941012d0 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5941aa280 con 0x7fb594100920 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59b2b3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5941aa410 con 0x7fb5941070f0 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52702/0 (socket says 172.21.15.121:52702) 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 -- 172.21.15.121:0/2085849225 learned_addr learned my addr 172.21.15.121:0/2085849225 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb599850700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941070f0 0x7fb594102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59904f700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 0x7fb5941018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:38.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 -- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941070f0 msgr2=0x7fb594102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941070f0 0x7fb594102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 -- 172.21.15.121:0/2085849225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 msgr2=0x7fb5941018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 0x7fb5941018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59884e700 1 -- 172.21.15.121:0/2085849225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584009520 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.919+0000 7fb59904f700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 0x7fb5941018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb59884e700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb58400ea30 tx=0x7fb58400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584009820 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb5840044a0 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5900095f0 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb584003ca0 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.920+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941aaa60 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.921+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594069f40 con 0x7fb5941012d0 2024-02-10T05:58:38.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.921+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fb584003e20 con 0x7fb5941012d0 2024-02-10T05:58:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.922+0000 7fb58a7fc700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 0x7fb58006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.922+0000 7fb59904f700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 0x7fb58006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.922+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fb584014070 con 0x7fb5941012d0 2024-02-10T05:58:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.922+0000 7fb59904f700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 0x7fb58006ff80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5900059d0 tx=0x7fb590005960 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:38.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:38.926+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb584056ff0 con 0x7fb5941012d0 2024-02-10T05:58:39.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.127+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb594102dd0 con 0x7fb5941012d0 2024-02-10T05:58:39.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.127+0000 7fb58a7fc700 1 -- 172.21.15.121:0/2085849225 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb58405a6b0 con 0x7fb5941012d0 2024-02-10T05:58:39.128 INFO:teuthology.orchestra.run.smithi121.stdout:77309411372 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 msgr2=0x7fb58006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 0x7fb58006ff80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5900059d0 tx=0x7fb590005960 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 msgr2=0x7fb594101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb58400ea30 tx=0x7fb58400ed40 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 shutdown_connections 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006dac0 0x7fb58006ff80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5941070f0 0x7fb594102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594100920 0x7fb5941018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 --2- 172.21.15.121:0/2085849225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5941012d0 0x7fb594101de0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 >> 172.21.15.121:0/2085849225 conn(0x7fb594075fa0 msgr2=0x7fb5940fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 shutdown_connections 2024-02-10T05:58:39.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.129+0000 7fb59b2b3700 1 -- 172.21.15.121:0/2085849225 wait complete. 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 -- 172.21.15.121:0/3398898160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 msgr2=0x7fb86410eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/3398898160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86410eb70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00b0a0 tx=0x7fb85c00b3b0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 -- 172.21.15.121:0/3398898160 shutdown_connections 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/3398898160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86410eb70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/3398898160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864107d50 0x7fb864108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/3398898160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864101d70 0x7fb8641021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 -- 172.21.15.121:0/3398898160 >> 172.21.15.121:0/3398898160 conn(0x7fb864078e50 msgr2=0x7fb864079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 -- 172.21.15.121:0/3398898160 shutdown_connections 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 -- 172.21.15.121:0/3398898160 wait complete. 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.581+0000 7fb86b5f0700 1 Processor -- start 2024-02-10T05:58:39.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 -- start start 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 0x7fb86419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 0x7fb864197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864197e00 con 0x7fb864107d50 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864197f70 con 0x7fb864101d70 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86b5f0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb864198130 con 0x7fb864102730 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb868b8b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb868b8b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:51992/0 (socket says 172.21.15.121:51992) 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 0x7fb864197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:39.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb868b8b700 1 -- 172.21.15.121:0/2159656320 learned_addr learned my addr 172.21.15.121:0/2159656320 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86938c700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 0x7fb86419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 -- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 msgr2=0x7fb86419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 -- 172.21.15.121:0/2159656320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 msgr2=0x7fb86419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 0x7fb86419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 -- 172.21.15.121:0/2159656320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854009520 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb86938c700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 0x7fb86419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb868b8b700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.582+0000 7fb869b8d700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 0x7fb864197650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00b070 tx=0x7fb85c0078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.583+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c00e040 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.583+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c009d00 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.583+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb85c01d470 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.583+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c0052a0 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.583+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8641a4330 con 0x7fb864107d50 2024-02-10T05:58:39.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.584+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb864103400 con 0x7fb864107d50 2024-02-10T05:58:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.584+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fb85c004a50 con 0x7fb864107d50 2024-02-10T05:58:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.585+0000 7fb85a7fc700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 0x7fb85006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.585+0000 7fb86938c700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 0x7fb85006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.585+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fb85c08ed70 con 0x7fb864107d50 2024-02-10T05:58:39.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.585+0000 7fb86938c700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 0x7fb85006ff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb8600099d0 tx=0x7fb860008040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:39.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.589+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb85c058180 con 0x7fb864107d50 2024-02-10T05:58:39.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2085849225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:58:39.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.785+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb864198b80 con 0x7fb864107d50 2024-02-10T05:58:39.786 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2085849225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:58:39.786 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2085849225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:58:39.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.785+0000 7fb85a7fc700 1 -- 172.21.15.121:0/2159656320 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb85c02e350 con 0x7fb864107d50 2024-02-10T05:58:39.787 INFO:teuthology.orchestra.run.smithi121.stdout:111669149735 2024-02-10T05:58:39.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 msgr2=0x7fb85006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 0x7fb85006ff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb8600099d0 tx=0x7fb860008040 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 msgr2=0x7fb864197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 0x7fb864197650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00b070 tx=0x7fb85c0078e0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 shutdown_connections 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb85006dac0 0x7fb85006ff80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb864107d50 0x7fb864197650 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb864101d70 0x7fb86419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 --2- 172.21.15.121:0/2159656320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb864102730 0x7fb86419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 >> 172.21.15.121:0/2159656320 conn(0x7fb864078e50 msgr2=0x7fb864105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 shutdown_connections 2024-02-10T05:58:39.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:39.788+0000 7fb86b5f0700 1 -- 172.21.15.121:0/2159656320 wait complete. 2024-02-10T05:58:39.942 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-02-10T05:58:39.942 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:40.485 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:40 smithi121 ceph-mon[41707]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:40.485 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2159656320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:58:40.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:40 smithi121 ceph-mon[30020]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:40.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2159656320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:58:40.515 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149732 got 111669149735 for osd.3 2024-02-10T05:58:40.515 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:40 smithi146 ceph-mon[33795]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2159656320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- 172.21.15.121:0/2124079799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e341018a0 msgr2=0x7f3e341058f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 --2- 172.21.15.121:0/2124079799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e341018a0 0x7f3e341058f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0098f0 tx=0x7f3e1c009c00 comp rx=0 tx=0).stop 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- 172.21.15.121:0/2124079799 shutdown_connections 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 --2- 172.21.15.121:0/2124079799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34105fd0 0x7f3e3410dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 --2- 172.21.15.121:0/2124079799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e34100ef0 0x7f3e341012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 --2- 172.21.15.121:0/2124079799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e341018a0 0x7f3e341058f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- 172.21.15.121:0/2124079799 >> 172.21.15.121:0/2124079799 conn(0x7f3e34077d50 msgr2=0x7f3e34078160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- 172.21.15.121:0/2124079799 shutdown_connections 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- 172.21.15.121:0/2124079799 wait complete. 2024-02-10T05:58:41.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 Processor -- start 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 -- start start 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.127+0000 7f3e39270700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 0x7f3e3419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 0x7f3e34197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e34197df0 con 0x7f3e341018a0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e34197f60 con 0x7f3e34100ef0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e34198120 con 0x7f3e34105fd0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52742/0 (socket says 172.21.15.121:52742) 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 -- 172.21.15.121:0/2867541788 learned_addr learned my addr 172.21.15.121:0/2867541788 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e337fe700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 0x7f3e34197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e32ffd700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 0x7f3e3419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 -- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 msgr2=0x7f3e34197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 0x7f3e34197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 -- 172.21.15.121:0/2867541788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 msgr2=0x7f3e3419d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 0x7f3e3419d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 -- 172.21.15.121:0/2867541788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240095f0 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e32ffd700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 0x7f3e3419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e337fe700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 0x7f3e34197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e2bfff700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009230 tx=0x7f3e1c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c01c070 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1c009520 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3e1c00bb60 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0038e0 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.128+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e341a4320 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.129+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e341023d0 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.130+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f3e1c020030 con 0x7f3e341018a0 2024-02-10T05:58:41.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.130+0000 7f3e30ff9700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 0x7f3e2006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.130+0000 7f3e32ffd700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 0x7f3e2006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.130+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f3e1c08e8f0 con 0x7f3e341018a0 2024-02-10T05:58:41.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.130+0000 7f3e32ffd700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 0x7f3e2006ff80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005fd0 tx=0x7f3e24009310 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.135+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e1c057d00 con 0x7f3e341018a0 2024-02-10T05:58:41.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 -- 172.21.15.121:0/3254506107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c0699d0 msgr2=0x7f881c069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 --2- 172.21.15.121:0/3254506107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c0699d0 0x7f881c069e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8810009920 tx=0x7f8810009c30 comp rx=0 tx=0).stop 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 -- 172.21.15.121:0/3254506107 shutdown_connections 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 --2- 172.21.15.121:0/3254506107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c06a370 0x7f881c10ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 --2- 172.21.15.121:0/3254506107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 0x7f881c108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 --2- 172.21.15.121:0/3254506107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c0699d0 0x7f881c069e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 -- 172.21.15.121:0/3254506107 >> 172.21.15.121:0/3254506107 conn(0x7f881c077b50 msgr2=0x7f881c077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 -- 172.21.15.121:0/3254506107 shutdown_connections 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 -- 172.21.15.121:0/3254506107 wait complete. 2024-02-10T05:58:41.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.328+0000 7f88212db700 1 Processor -- start 2024-02-10T05:58:41.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 -- start start 2024-02-10T05:58:41.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 0x7f881c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 0x7f881c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 0x7f881c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c197c80 con 0x7f881c06a370 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c197dc0 con 0x7f881c107d70 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f88212db700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c197f80 con 0x7f881c0699d0 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 0x7f881c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 0x7f881c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 0x7f881c19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52768/0 (socket says 172.21.15.121:52768) 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881a7fc700 1 -- 172.21.15.121:0/4266972216 learned_addr learned my addr 172.21.15.121:0/4266972216 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881b7fe700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 0x7f881c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 -- 172.21.15.121:0/4266972216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 msgr2=0x7f881c197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 0x7f881c197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 -- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 msgr2=0x7f881c19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.332 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 0x7f881c19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.329+0000 7f881affd700 1 -- 172.21.15.121:0/4266972216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814009d00 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f881affd700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 0x7f881c19d460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f880c00d8d0 tx=0x7f880c00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c00f890 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f880c00fe70 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8810009520 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c100c20 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.330+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880c019360 con 0x7f881c0699d0 2024-02-10T05:58:41.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.331+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881c10a290 con 0x7f881c0699d0 2024-02-10T05:58:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.332+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f880c00f9f0 con 0x7f881c0699d0 2024-02-10T05:58:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.332+0000 7f8803fff700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 0x7f880406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.332+0000 7f881a7fc700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 0x7f880406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.332+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f880c08d490 con 0x7f881c0699d0 2024-02-10T05:58:41.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.333+0000 7f881a7fc700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 0x7f880406ff80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f881000c010 tx=0x7f8810019040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.340+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3e34198b70 con 0x7f3e341018a0 2024-02-10T05:58:41.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.340+0000 7f3e30ff9700 1 -- 172.21.15.121:0/2867541788 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3e1c05b3c0 con 0x7f3e341018a0 2024-02-10T05:58:41.340 INFO:teuthology.orchestra.run.smithi121.stdout:167503724572 2024-02-10T05:58:41.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.341+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f880c056920 con 0x7f881c0699d0 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 msgr2=0x7f3e2006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 0x7f3e2006ff80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005fd0 tx=0x7f3e24009310 comp rx=0 tx=0).stop 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 msgr2=0x7f3e3419dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009230 tx=0x7f3e1c00bf20 comp rx=0 tx=0).stop 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 shutdown_connections 2024-02-10T05:58:41.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f3e2006dac0 0x7f3e2006ff80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e34105fd0 0x7f3e34197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e34100ef0 0x7f3e3419d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 --2- 172.21.15.121:0/2867541788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e341018a0 0x7f3e3419dab0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 >> 172.21.15.121:0/2867541788 conn(0x7f3e34077d50 msgr2=0x7f3e341071a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 shutdown_connections 2024-02-10T05:58:41.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.343+0000 7f3e39270700 1 -- 172.21.15.121:0/2867541788 wait complete. 2024-02-10T05:58:41.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.544+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f881c0689e0 con 0x7f881c0699d0 2024-02-10T05:58:41.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.544+0000 7f8803fff700 1 -- 172.21.15.121:0/4266972216 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f880c059fe0 con 0x7f881c0699d0 2024-02-10T05:58:41.545 INFO:teuthology.orchestra.run.smithi121.stdout:34359738423 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.546+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 msgr2=0x7f880406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.546+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 0x7f880406ff80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f881000c010 tx=0x7f8810019040 comp rx=0 tx=0).stop 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.546+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 msgr2=0x7f881c19d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.546+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 0x7f881c19d460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f880c00d8d0 tx=0x7f880c00dbe0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.546+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 shutdown_connections 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f880406dac0 0x7f880406ff80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f881c107d70 0x7f881c197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f881c0699d0 0x7f881c19d460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 --2- 172.21.15.121:0/4266972216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f881c06a370 0x7f881c19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 >> 172.21.15.121:0/4266972216 conn(0x7f881c077b50 msgr2=0x7f881c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 shutdown_connections 2024-02-10T05:58:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.547+0000 7f88212db700 1 -- 172.21.15.121:0/4266972216 wait complete. 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 -- 172.21.15.121:0/1318394499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0101da0 msgr2=0x7f7fd0102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/1318394499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0101da0 0x7f7fd0102220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009990 tx=0x7f7fb8009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 -- 172.21.15.121:0/1318394499 shutdown_connections 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/1318394499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0102760 0x7f7fd010ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/1318394499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0107e80 0x7f7fd0108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/1318394499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0101da0 0x7f7fd0102220 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.571+0000 7f7fd5a52700 1 -- 172.21.15.121:0/1318394499 >> 172.21.15.121:0/1318394499 conn(0x7f7fd0078de0 msgr2=0x7f7fd00791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.572+0000 7f7fd5a52700 1 -- 172.21.15.121:0/1318394499 shutdown_connections 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.572+0000 7f7fd5a52700 1 -- 172.21.15.121:0/1318394499 wait complete. 2024-02-10T05:58:41.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.572+0000 7f7fd5a52700 1 Processor -- start 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.572+0000 7f7fd5a52700 1 -- start start 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0101da0 0x7f7fd019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0197c70 con 0x7f7fd0107e80 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0197db0 con 0x7f7fd0101da0 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd0197f70 con 0x7f7fd0102760 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fcf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52052/0 (socket says 172.21.15.121:52052) 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fcf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52786/0 (socket says 172.21.15.121:52786) 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 -- 172.21.15.121:0/3524023235 learned_addr learned my addr 172.21.15.121:0/3524023235 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fceffd700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0101da0 0x7f7fd019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 -- 172.21.15.121:0/3524023235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0101da0 msgr2=0x7f7fd019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0101da0 0x7f7fd019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 -- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 msgr2=0x7f7fd0197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 -- 172.21.15.121:0/3524023235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8009c40 con 0x7f7fd0102760 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fcf7fe700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fc7fff700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8006010 tx=0x7f7fb80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb801d070 con 0x7f7fd0102760 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb80095f0 con 0x7f7fd0102760 2024-02-10T05:58:41.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7fb800bdc0 con 0x7f7fd0102760 2024-02-10T05:58:41.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.573+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8022d50 con 0x7f7fd0102760 2024-02-10T05:58:41.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.574+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd01a4320 con 0x7f7fd0102760 2024-02-10T05:58:41.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.575+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd00689e0 con 0x7f7fd0102760 2024-02-10T05:58:41.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.575+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f7fb800b940 con 0x7f7fd0102760 2024-02-10T05:58:41.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.576+0000 7f7fd4a50700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 0x7f7fbc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:41.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.576+0000 7f7fceffd700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 0x7f7fbc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:41.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.576+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f7fb808ef00 con 0x7f7fd0102760 2024-02-10T05:58:41.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.576+0000 7f7fceffd700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 0x7f7fbc06ff80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0006fd0 tx=0x7f7fc0008040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:41.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.581+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7fb8058390 con 0x7f7fd0102760 2024-02-10T05:58:41.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2867541788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:58:41.780 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2867541788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:58:41.781 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2867541788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:58:41.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.780+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7fd0198130 con 0x7f7fd0102760 2024-02-10T05:58:41.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.780+0000 7f7fd4a50700 1 -- 172.21.15.121:0/3524023235 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7fb8027090 con 0x7f7fd0102760 2024-02-10T05:58:41.781 INFO:teuthology.orchestra.run.smithi121.stdout:201863462936 2024-02-10T05:58:41.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.783+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 msgr2=0x7f7fbc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.783+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 0x7f7fbc06ff80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0006fd0 tx=0x7f7fc0008040 comp rx=0 tx=0).stop 2024-02-10T05:58:41.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.783+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 msgr2=0x7f7fd019d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:41.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.783+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8006010 tx=0x7f7fb80057f0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 shutdown_connections 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7fbc06dac0 0x7f7fbc06ff80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fd0107e80 0x7f7fd0197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fd0101da0 0x7f7fd019d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 --2- 172.21.15.121:0/3524023235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fd0102760 0x7f7fd019d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 >> 172.21.15.121:0/3524023235 conn(0x7f7fd0078de0 msgr2=0x7f7fd0105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 shutdown_connections 2024-02-10T05:58:41.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:41.784+0000 7f7fd5a52700 1 -- 172.21.15.121:0/3524023235 wait complete. 2024-02-10T05:58:42.209 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738423 for osd.0 2024-02-10T05:58:42.209 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:42.527 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[30020]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:42.527 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4266972216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:58:42.528 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3524023235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:58:42.528 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[41707]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:42.528 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4266972216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:58:42.528 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3524023235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:58:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:42 smithi146 ceph-mon[33795]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4266972216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:58:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3524023235' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:58:43.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 -- 172.21.15.121:0/877906916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0100eb0 msgr2=0x7f23f0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 --2- 172.21.15.121:0/877906916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0100eb0 0x7f23f0101290 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009990 tx=0x7f23ec009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 -- 172.21.15.121:0/877906916 shutdown_connections 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 --2- 172.21.15.121:0/877906916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 --2- 172.21.15.121:0/877906916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0100eb0 0x7f23f0101290 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 --2- 172.21.15.121:0/877906916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0101860 0x7f23f0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 -- 172.21.15.121:0/877906916 >> 172.21.15.121:0/877906916 conn(0x7f23f0076c70 msgr2=0x7f23f0077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 -- 172.21.15.121:0/877906916 shutdown_connections 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.001+0000 7f23f8311700 1 -- 172.21.15.121:0/877906916 wait complete. 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 Processor -- start 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 -- start start 2024-02-10T05:58:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 0x7f23f0196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f019b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f019b620 con 0x7f23f0101860 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f019b790 con 0x7f23f0105f00 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f8311700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f019b950 con 0x7f23f0100eb0 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f60ad700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 0x7f23f0196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f60ad700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 0x7f23f0196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52066/0 (socket says 172.21.15.121:52066) 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52810/0 (socket says 172.21.15.121:52810) 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f60ad700 1 -- 172.21.15.121:0/31535650 learned_addr learned my addr 172.21.15.121:0/31535650 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f68ae700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f019b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 -- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 msgr2=0x7f23f0196740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 0x7f23f0196740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 -- 172.21.15.121:0/31535650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 msgr2=0x7f23f019b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f019b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 -- 172.21.15.121:0/31535650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e0009520 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f68ae700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f019b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.002+0000 7f23f58ac700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23e000e9c0 tx=0x7f23e000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.003+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e000cb70 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.003+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f23e00044a0 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.003+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec0095f0 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.003+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0003cb0 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.003+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f01a32c0 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.004+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f00689e0 con 0x7f23f0101860 2024-02-10T05:58:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.004+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f23e0003e10 con 0x7f23f0101860 2024-02-10T05:58:43.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.004+0000 7f23e77fe700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 0x7f23dc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:43.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.005+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f23e0014070 con 0x7f23f0101860 2024-02-10T05:58:43.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.005+0000 7f23f60ad700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 0x7f23dc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:43.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.005+0000 7f23f60ad700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 0x7f23dc06ff80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec005400 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.010+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f23e0056fb0 con 0x7f23f0101860 2024-02-10T05:58:43.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.211+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f23f0110180 con 0x7f23f0101860 2024-02-10T05:58:43.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.211+0000 7f23e77fe700 1 -- 172.21.15.121:0/31535650 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f23e005a670 con 0x7f23f0101860 2024-02-10T05:58:43.212 INFO:teuthology.orchestra.run.smithi121.stdout:55834574897 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 msgr2=0x7f23dc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 0x7f23dc06ff80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec005400 comp rx=0 tx=0).stop 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 msgr2=0x7f23f0196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f23e000e9c0 tx=0x7f23e000ecd0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 shutdown_connections 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f23dc06dac0 0x7f23dc06ff80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23f0105f00 0x7f23f019b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f23f0100eb0 0x7f23f0196740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 --2- 172.21.15.121:0/31535650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23f0101860 0x7f23f0196c80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.213+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 >> 172.21.15.121:0/31535650 conn(0x7f23f0076c70 msgr2=0x7f23f00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.214+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 shutdown_connections 2024-02-10T05:58:43.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:43.214+0000 7f23f8311700 1 -- 172.21.15.121:0/31535650 wait complete. 2024-02-10T05:58:43.458 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:43.484 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724569 got 167503724572 for osd.5 2024-02-10T05:58:43.484 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[30020]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/31535650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[41707]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:44.055 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/31535650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:58:44.056 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.056 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.334 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574897 for osd.1 2024-02-10T05:58:44.334 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:44.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:44 smithi146 ceph-mon[33795]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/31535650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:58:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:44.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462932 got 201863462936 for osd.6 2024-02-10T05:58:44.693 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:45.280 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:58:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:58:45.281 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:45 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:45.281 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:45 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:45 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:58:46.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:45 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.204 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:46.205 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.218+0000 7f5511740700 1 -- 172.21.15.121:0/3675206832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c100eb0 msgr2=0x7f550c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.218+0000 7f5511740700 1 --2- 172.21.15.121:0/3675206832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c100eb0 0x7f550c101290 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f54fc009990 tx=0x7f54fc009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.218+0000 7f5511740700 1 -- 172.21.15.121:0/3675206832 shutdown_connections 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.218+0000 7f5511740700 1 --2- 172.21.15.121:0/3675206832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 --2- 172.21.15.121:0/3675206832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c100eb0 0x7f550c101290 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 --2- 172.21.15.121:0/3675206832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c101860 0x7f550c105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 -- 172.21.15.121:0/3675206832 >> 172.21.15.121:0/3675206832 conn(0x7f550c076c70 msgr2=0x7f550c077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:46.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 -- 172.21.15.121:0/3675206832 shutdown_connections 2024-02-10T05:58:46.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 -- 172.21.15.121:0/3675206832 wait complete. 2024-02-10T05:58:46.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.219+0000 7f5511740700 1 Processor -- start 2024-02-10T05:58:46.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 -- start start 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 0x7f550c198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c1a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c199440 con 0x7f550c101860 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1a3c20 con 0x7f550c100eb0 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f5511740700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550c1a3d60 con 0x7f550c105f00 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52832/0 (socket says 172.21.15.121:52832) 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 -- 172.21.15.121:0/868283625 learned_addr learned my addr 172.21.15.121:0/868283625 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550affd700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 0x7f550c198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550b7fe700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c1a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:46.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 -- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 msgr2=0x7f550c1a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c1a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 -- 172.21.15.121:0/868283625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 msgr2=0x7f550c198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 0x7f550c198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550a7fc700 1 -- 172.21.15.121:0/868283625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5504009c40 con 0x7f550c101860 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550b7fe700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c1a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.220+0000 7f550affd700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 0x7f550c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f550a7fc700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f550000e9a0 tx=0x7f550000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500009790 con 0x7f550c101860 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5500005020 con 0x7f550c101860 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc0095f0 con 0x7f550c101860 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000cba0 con 0x7f550c101860 2024-02-10T05:58:46.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.221+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c1a42f0 con 0x7f550c101860 2024-02-10T05:58:46.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.222+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550c1091c0 con 0x7f550c101860 2024-02-10T05:58:46.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.222+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f550000cd00 con 0x7f550c101860 2024-02-10T05:58:46.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.223+0000 7f54f3fff700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 0x7f54f406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:46.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.223+0000 7f550affd700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 0x7f54f406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:46.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.223+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f5500014070 con 0x7f550c101860 2024-02-10T05:58:46.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.223+0000 7f550affd700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 0x7f54f406ff80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b2e0 tx=0x7f54fc005040 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:46.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.228+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5500056fd0 con 0x7f550c101860 2024-02-10T05:58:46.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:58:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:58:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.438+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f550c0689e0 con 0x7f550c101860 2024-02-10T05:58:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.438+0000 7f54f3fff700 1 -- 172.21.15.121:0/868283625 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f550005a690 con 0x7f550c101860 2024-02-10T05:58:46.439 INFO:teuthology.orchestra.run.smithi121.stdout:240518168595 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 msgr2=0x7f54f406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 0x7f54f406ff80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00b2e0 tx=0x7f54fc005040 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 msgr2=0x7f550c198ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f550000e9a0 tx=0x7f550000ecb0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 shutdown_connections 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f54f406dac0 0x7f54f406ff80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f550c105f00 0x7f550c1a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f550c100eb0 0x7f550c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 --2- 172.21.15.121:0/868283625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f550c101860 0x7f550c198ba0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 >> 172.21.15.121:0/868283625 conn(0x7f550c076c70 msgr2=0x7f550c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 shutdown_connections 2024-02-10T05:58:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:46.441+0000 7f5511740700 1 -- 172.21.15.121:0/868283625 wait complete. 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 3760M 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.210 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/868283625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 3760M 2024-02-10T05:58:47.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/868283625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.234 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168592 got 240518168595 for osd.7 2024-02-10T05:58:47.235 DEBUG:teuthology.parallel:result is None 2024-02-10T05:58:47.235 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T05:58:47.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:58:47.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 3760M 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi121:/etc/ceph/ceph.conf 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi146:/etc/ceph/ceph.conf 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: Updating smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/868283625' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:47.495 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:48.327 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:48 smithi121 ceph-mon[41707]: Deploying daemon alertmanager.a on smithi121 2024-02-10T05:58:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:48 smithi121 ceph-mon[41707]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:48.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:48 smithi121 ceph-mon[30020]: Deploying daemon alertmanager.a on smithi121 2024-02-10T05:58:48.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:48 smithi121 ceph-mon[30020]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:48.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:48 smithi146 ceph-mon[33795]: Deploying daemon alertmanager.a on smithi121 2024-02-10T05:58:48.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:48 smithi146 ceph-mon[33795]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:49.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 -- 172.21.15.121:0/2433342220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 msgr2=0x7f5fa410d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:49.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 --2- 172.21.15.121:0/2433342220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa410d2b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009990 tx=0x7f5f94009ca0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 -- 172.21.15.121:0/2433342220 shutdown_connections 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 --2- 172.21.15.121:0/2433342220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa410d2b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 --2- 172.21.15.121:0/2433342220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa4106470 0x7f5fa4106850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 --2- 172.21.15.121:0/2433342220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa41003e0 0x7f5fa4100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 -- 172.21.15.121:0/2433342220 >> 172.21.15.121:0/2433342220 conn(0x7f5fa4077220 msgr2=0x7f5fa40fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 -- 172.21.15.121:0/2433342220 shutdown_connections 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.557+0000 7f5fa9216700 1 -- 172.21.15.121:0/2433342220 wait complete. 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 Processor -- start 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 -- start start 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa41003e0 0x7f5fa419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:49.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa419daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4197de0 con 0x7f5fa4106470 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4197f50 con 0x7f5fa41003e0 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa9216700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa4198110 con 0x7f5fa4100da0 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa259c700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa419daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50574/0 (socket says 172.21.15.121:50574) 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa359e700 1 -- 172.21.15.121:0/1760501940 learned_addr learned my addr 172.21.15.121:0/1760501940 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.558+0000 7f5fa2d9d700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa41003e0 0x7f5fa419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 -- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 msgr2=0x7f5fa419daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa419daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 -- 172.21.15.121:0/1760501940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa41003e0 msgr2=0x7f5fa419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:49.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa41003e0 0x7f5fa419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 -- 172.21.15.121:0/1760501940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009520 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa259c700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa359e700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098e0 tx=0x7f5f94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9401d070 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f940095f0 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5f94021410 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f940173f0 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.559+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa41a4310 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.560+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa4101b20 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.561+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f5f9400fb10 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.561+0000 7f5f8bfff700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 0x7f5f8c06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.561+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f5f9408eb90 con 0x7f5fa4106470 2024-02-10T05:58:49.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.561+0000 7f5fa2d9d700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 0x7f5f8c06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:49.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.561+0000 7f5fa2d9d700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 0x7f5f8c06ff80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009f20 tx=0x7f5f9c0094f0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:49.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.566+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f94058170 con 0x7f5fa4106470 2024-02-10T05:58:49.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.761+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 --> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fa4111320 con 0x7f5f8c06dac0 2024-02-10T05:58:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.784+0000 7f5f8bfff700 1 -- 172.21.15.121:0/1760501940 <== mgr.24455 v2:172.21.15.121:6800/1142815253 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384365 (secure 0 0 0) 0x7f5fa4111320 con 0x7f5f8c06dac0 2024-02-10T05:58:49.789 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:58:49.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.787+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 msgr2=0x7f5f8c06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:49.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.787+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 0x7f5f8c06ff80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009f20 tx=0x7f5f9c0094f0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.787+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 msgr2=0x7f5fa4197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.787+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098e0 tx=0x7f5f94004730 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 shutdown_connections 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f5f8c06dac0 0x7f5f8c06ff80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fa4106470 0x7f5fa4197630 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fa41003e0 0x7f5fa419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 --2- 172.21.15.121:0/1760501940 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fa4100da0 0x7f5fa419daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 >> 172.21.15.121:0/1760501940 conn(0x7f5fa4077220 msgr2=0x7f5fa4102e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 shutdown_connections 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:49.788+0000 7f5fa9216700 1 -- 172.21.15.121:0/1760501940 wait complete. 2024-02-10T05:58:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-10T05:58:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:50 smithi121 ceph-mon[41707]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:50 smithi121 ceph-mon[30020]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:50.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:50 smithi146 ceph-mon[33795]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:51 smithi121 ceph-mon[41707]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:51.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:51 smithi121 ceph-mon[30020]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:51.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:51 smithi146 ceph-mon[33795]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:51.452 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-02-10T05:58:48.917405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1288,"num_read_kb":1102,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":798,"ondisk_log_size":798,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236888,"kb_used_data":8252,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724328,"statfs":{"total":767960285184,"available":767717711872,"internally_reserved":0,"allocated":8450048,"data_stored":4545620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":234016307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002912"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048473+0000","last_change":"2024-02-10T05:57:38.900690+0000","last_active":"2024-02-10T05:58:15.048473+0000","last_peered":"2024-02-10T05:58:15.048473+0000","last_clean":"2024-02-10T05:58:15.048473+0000","last_became_active":"2024-02-10T05:57:38.900430+0000","last_became_peered":"2024-02-10T05:57:38.900430+0000","last_unstale":"2024-02-10T05:58:15.048473+0000","last_undegraded":"2024-02-10T05:58:15.048473+0000","last_fullsized":"2024-02-10T05:58:15.048473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:19.909192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901193+0000","last_change":"2024-02-10T05:57:36.907884+0000","last_active":"2024-02-10T05:58:14.901193+0000","last_peered":"2024-02-10T05:58:14.901193+0000","last_clean":"2024-02-10T05:58:14.901193+0000","last_became_active":"2024-02-10T05:57:36.907441+0000","last_became_peered":"2024-02-10T05:57:36.907441+0000","last_unstale":"2024-02-10T05:58:14.901193+0000","last_undegraded":"2024-02-10T05:58:14.901193+0000","last_fullsized":"2024-02-10T05:58:14.901193+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:12:27.847516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368017+0000","last_change":"2024-02-10T05:57:33.145239+0000","last_active":"2024-02-10T05:58:15.368017+0000","last_peered":"2024-02-10T05:58:15.368017+0000","last_clean":"2024-02-10T05:58:15.368017+0000","last_became_active":"2024-02-10T05:57:33.144907+0000","last_became_peered":"2024-02-10T05:57:33.144907+0000","last_unstale":"2024-02-10T05:58:15.368017+0000","last_undegraded":"2024-02-10T05:58:15.368017+0000","last_fullsized":"2024-02-10T05:58:15.368017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:13.513400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"70'16","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048383+0000","last_change":"2024-02-10T05:57:34.881439+0000","last_active":"2024-02-10T05:58:15.048383+0000","last_peered":"2024-02-10T05:58:15.048383+0000","last_clean":"2024-02-10T05:58:15.048383+0000","last_became_active":"2024-02-10T05:57:34.881048+0000","last_became_peered":"2024-02-10T05:57:34.881048+0000","last_unstale":"2024-02-10T05:58:15.048383+0000","last_undegraded":"2024-02-10T05:58:15.048383+0000","last_fullsized":"2024-02-10T05:58:15.048383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:24:04.210294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903287+0000","last_change":"2024-02-10T05:57:38.949010+0000","last_active":"2024-02-10T05:58:14.903287+0000","last_peered":"2024-02-10T05:58:14.903287+0000","last_clean":"2024-02-10T05:58:14.903287+0000","last_became_active":"2024-02-10T05:57:38.948657+0000","last_became_peered":"2024-02-10T05:57:38.948657+0000","last_unstale":"2024-02-10T05:58:14.903287+0000","last_undegraded":"2024-02-10T05:58:14.903287+0000","last_fullsized":"2024-02-10T05:58:14.903287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:43:15.197929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048295+0000","last_change":"2024-02-10T05:57:36.902625+0000","last_active":"2024-02-10T05:58:15.048295+0000","last_peered":"2024-02-10T05:58:15.048295+0000","last_clean":"2024-02-10T05:58:15.048295+0000","last_became_active":"2024-02-10T05:57:36.902344+0000","last_became_peered":"2024-02-10T05:57:36.902344+0000","last_unstale":"2024-02-10T05:58:15.048295+0000","last_undegraded":"2024-02-10T05:58:15.048295+0000","last_fullsized":"2024-02-10T05:58:15.048295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:01.464104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902819+0000","last_change":"2024-02-10T05:57:33.145205+0000","last_active":"2024-02-10T05:58:14.902819+0000","last_peered":"2024-02-10T05:58:14.902819+0000","last_clean":"2024-02-10T05:58:14.902819+0000","last_became_active":"2024-02-10T05:57:33.144875+0000","last_became_peered":"2024-02-10T05:57:33.144875+0000","last_unstale":"2024-02-10T05:58:14.902819+0000","last_undegraded":"2024-02-10T05:58:14.902819+0000","last_fullsized":"2024-02-10T05:58:14.902819+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:51:17.689770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"70'39","reported_seq":102,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903063+0000","last_change":"2024-02-10T05:57:34.878160+0000","last_active":"2024-02-10T05:58:14.903063+0000","last_peered":"2024-02-10T05:58:14.903063+0000","last_clean":"2024-02-10T05:58:14.903063+0000","last_became_active":"2024-02-10T05:57:34.877851+0000","last_became_peered":"2024-02-10T05:57:34.877851+0000","last_unstale":"2024-02-10T05:58:14.903063+0000","last_undegraded":"2024-02-10T05:58:14.903063+0000","last_fullsized":"2024-02-10T05:58:14.903063+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:20:23.895171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367731+0000","last_change":"2024-02-10T05:57:38.914265+0000","last_active":"2024-02-10T05:58:15.367731+0000","last_peered":"2024-02-10T05:58:15.367731+0000","last_clean":"2024-02-10T05:58:15.367731+0000","last_became_active":"2024-02-10T05:57:38.913956+0000","last_became_peered":"2024-02-10T05:57:38.913956+0000","last_unstale":"2024-02-10T05:58:15.367731+0000","last_undegraded":"2024-02-10T05:58:15.367731+0000","last_fullsized":"2024-02-10T05:58:15.367731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:36:01.753839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006068+0000","last_change":"2024-02-10T05:57:36.903461+0000","last_active":"2024-02-10T05:58:15.006068+0000","last_peered":"2024-02-10T05:58:15.006068+0000","last_clean":"2024-02-10T05:58:15.006068+0000","last_became_active":"2024-02-10T05:57:36.903183+0000","last_became_peered":"2024-02-10T05:57:36.903183+0000","last_unstale":"2024-02-10T05:58:15.006068+0000","last_undegraded":"2024-02-10T05:58:15.006068+0000","last_fullsized":"2024-02-10T05:58:15.006068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:51:01.700375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006192+0000","last_change":"2024-02-10T05:57:32.859268+0000","last_active":"2024-02-10T05:58:15.006192+0000","last_peered":"2024-02-10T05:58:15.006192+0000","last_clean":"2024-02-10T05:58:15.006192+0000","last_became_active":"2024-02-10T05:57:32.858544+0000","last_became_peered":"2024-02-10T05:57:32.858544+0000","last_unstale":"2024-02-10T05:58:15.006192+0000","last_undegraded":"2024-02-10T05:58:15.006192+0000","last_fullsized":"2024-02-10T05:58:15.006192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:59:58.445433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"70'21","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048633+0000","last_change":"2024-02-10T05:57:34.877523+0000","last_active":"2024-02-10T05:58:15.048633+0000","last_peered":"2024-02-10T05:58:15.048633+0000","last_clean":"2024-02-10T05:58:15.048633+0000","last_became_active":"2024-02-10T05:57:34.877220+0000","last_became_peered":"2024-02-10T05:57:34.877220+0000","last_unstale":"2024-02-10T05:58:15.048633+0000","last_undegraded":"2024-02-10T05:58:15.048633+0000","last_fullsized":"2024-02-10T05:58:15.048633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:19.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005819+0000","last_change":"2024-02-10T05:57:38.909792+0000","last_active":"2024-02-10T05:58:15.005819+0000","last_peered":"2024-02-10T05:58:15.005819+0000","last_clean":"2024-02-10T05:58:15.005819+0000","last_became_active":"2024-02-10T05:57:38.909424+0000","last_became_peered":"2024-02-10T05:57:38.909424+0000","last_unstale":"2024-02-10T05:58:15.005819+0000","last_undegraded":"2024-02-10T05:58:15.005819+0000","last_fullsized":"2024-02-10T05:58:15.005819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:12:23.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016394+0000","last_change":"2024-02-10T05:57:36.904444+0000","last_active":"2024-02-10T05:58:15.016394+0000","last_peered":"2024-02-10T05:58:15.016394+0000","last_clean":"2024-02-10T05:58:15.016394+0000","last_became_active":"2024-02-10T05:57:36.904176+0000","last_became_peered":"2024-02-10T05:57:36.904176+0000","last_unstale":"2024-02-10T05:58:15.016394+0000","last_undegraded":"2024-02-10T05:58:15.016394+0000","last_fullsized":"2024-02-10T05:58:15.016394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:28:39.123394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006222+0000","last_change":"2024-02-10T05:57:32.862310+0000","last_active":"2024-02-10T05:58:15.006222+0000","last_peered":"2024-02-10T05:58:15.006222+0000","last_clean":"2024-02-10T05:58:15.006222+0000","last_became_active":"2024-02-10T05:57:32.862010+0000","last_became_peered":"2024-02-10T05:57:32.862010+0000","last_unstale":"2024-02-10T05:58:15.006222+0000","last_undegraded":"2024-02-10T05:58:15.006222+0000","last_fullsized":"2024-02-10T05:58:15.006222+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:40:15.874141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"70'11","reported_seq":51,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048891+0000","last_change":"2024-02-10T05:57:34.881679+0000","last_active":"2024-02-10T05:58:15.048891+0000","last_peered":"2024-02-10T05:58:15.048891+0000","last_clean":"2024-02-10T05:58:15.048891+0000","last_became_active":"2024-02-10T05:57:34.881054+0000","last_became_peered":"2024-02-10T05:57:34.881054+0000","last_unstale":"2024-02-10T05:58:15.048891+0000","last_undegraded":"2024-02-10T05:58:15.048891+0000","last_fullsized":"2024-02-10T05:58:15.048891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:51.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049100+0000","last_change":"2024-02-10T05:57:38.946264+0000","last_active":"2024-02-10T05:58:15.049100+0000","last_peered":"2024-02-10T05:58:15.049100+0000","last_clean":"2024-02-10T05:58:15.049100+0000","last_became_active":"2024-02-10T05:57:38.945957+0000","last_became_peered":"2024-02-10T05:57:38.945957+0000","last_unstale":"2024-02-10T05:58:15.049100+0000","last_undegraded":"2024-02-10T05:58:15.049100+0000","last_fullsized":"2024-02-10T05:58:15.049100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:10.197651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902204+0000","last_change":"2024-02-10T05:57:36.901640+0000","last_active":"2024-02-10T05:58:14.902204+0000","last_peered":"2024-02-10T05:58:14.902204+0000","last_clean":"2024-02-10T05:58:14.902204+0000","last_became_active":"2024-02-10T05:57:36.901355+0000","last_became_peered":"2024-02-10T05:57:36.901355+0000","last_unstale":"2024-02-10T05:58:14.902204+0000","last_undegraded":"2024-02-10T05:58:14.902204+0000","last_fullsized":"2024-02-10T05:58:14.902204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:17:48.044556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368037+0000","last_change":"2024-02-10T05:57:32.861127+0000","last_active":"2024-02-10T05:58:15.368037+0000","last_peered":"2024-02-10T05:58:15.368037+0000","last_clean":"2024-02-10T05:58:15.368037+0000","last_became_active":"2024-02-10T05:57:32.860843+0000","last_became_peered":"2024-02-10T05:57:32.860843+0000","last_unstale":"2024-02-10T05:58:15.368037+0000","last_undegraded":"2024-02-10T05:58:15.368037+0000","last_fullsized":"2024-02-10T05:58:15.368037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:20:16.100213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"70'15","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013340+0000","last_change":"2024-02-10T05:57:34.879233+0000","last_active":"2024-02-10T05:58:15.013340+0000","last_peered":"2024-02-10T05:58:15.013340+0000","last_clean":"2024-02-10T05:58:15.013340+0000","last_became_active":"2024-02-10T05:57:34.878896+0000","last_became_peered":"2024-02-10T05:57:34.878896+0000","last_unstale":"2024-02-10T05:58:15.013340+0000","last_undegraded":"2024-02-10T05:58:15.013340+0000","last_fullsized":"2024-02-10T05:58:15.013340+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:08.661149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902161+0000","last_change":"2024-02-10T05:57:38.946919+0000","last_active":"2024-02-10T05:58:14.902161+0000","last_peered":"2024-02-10T05:58:14.902161+0000","last_clean":"2024-02-10T05:58:14.902161+0000","last_became_active":"2024-02-10T05:57:38.946234+0000","last_became_peered":"2024-02-10T05:57:38.946234+0000","last_unstale":"2024-02-10T05:58:14.902161+0000","last_undegraded":"2024-02-10T05:58:14.902161+0000","last_fullsized":"2024-02-10T05:58:14.902161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:17:02.563845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048868+0000","last_change":"2024-02-10T05:57:36.894935+0000","last_active":"2024-02-10T05:58:15.048868+0000","last_peered":"2024-02-10T05:58:15.048868+0000","last_clean":"2024-02-10T05:58:15.048868+0000","last_became_active":"2024-02-10T05:57:36.894624+0000","last_became_peered":"2024-02-10T05:57:36.894624+0000","last_unstale":"2024-02-10T05:58:15.048868+0000","last_undegraded":"2024-02-10T05:58:15.048868+0000","last_fullsized":"2024-02-10T05:58:15.048868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:21:36.311046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048673+0000","last_change":"2024-02-10T05:57:33.146228+0000","last_active":"2024-02-10T05:58:15.048673+0000","last_peered":"2024-02-10T05:58:15.048673+0000","last_clean":"2024-02-10T05:58:15.048673+0000","last_became_active":"2024-02-10T05:57:33.145911+0000","last_became_peered":"2024-02-10T05:57:33.145911+0000","last_unstale":"2024-02-10T05:58:15.048673+0000","last_undegraded":"2024-02-10T05:58:15.048673+0000","last_fullsized":"2024-02-10T05:58:15.048673+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:58:23.951660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"70'18","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902262+0000","last_change":"2024-02-10T05:57:34.885839+0000","last_active":"2024-02-10T05:58:14.902262+0000","last_peered":"2024-02-10T05:58:14.902262+0000","last_clean":"2024-02-10T05:58:14.902262+0000","last_became_active":"2024-02-10T05:57:34.885491+0000","last_became_peered":"2024-02-10T05:57:34.885491+0000","last_unstale":"2024-02-10T05:58:14.902262+0000","last_undegraded":"2024-02-10T05:58:14.902262+0000","last_fullsized":"2024-02-10T05:58:14.902262+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:11:54.147255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"70'1","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016633+0000","last_change":"2024-02-10T05:57:38.907926+0000","last_active":"2024-02-10T05:58:15.016633+0000","last_peered":"2024-02-10T05:58:15.016633+0000","last_clean":"2024-02-10T05:58:15.016633+0000","last_became_active":"2024-02-10T05:57:38.907637+0000","last_became_peered":"2024-02-10T05:57:38.907637+0000","last_unstale":"2024-02-10T05:58:15.016633+0000","last_undegraded":"2024-02-10T05:58:15.016633+0000","last_fullsized":"2024-02-10T05:58:15.016633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:07:49.741587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"70'8","reported_seq":51,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198649+0000","last_change":"2024-02-10T05:57:36.900385+0000","last_active":"2024-02-10T05:58:38.198649+0000","last_peered":"2024-02-10T05:58:38.198649+0000","last_clean":"2024-02-10T05:58:38.198649+0000","last_became_active":"2024-02-10T05:57:36.899506+0000","last_became_peered":"2024-02-10T05:57:36.899506+0000","last_unstale":"2024-02-10T05:58:38.198649+0000","last_undegraded":"2024-02-10T05:58:38.198649+0000","last_fullsized":"2024-02-10T05:58:38.198649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:27:39.093215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"63'1","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901278+0000","last_change":"2024-02-10T05:57:32.856157+0000","last_active":"2024-02-10T05:58:14.901278+0000","last_peered":"2024-02-10T05:58:14.901278+0000","last_clean":"2024-02-10T05:58:14.901278+0000","last_became_active":"2024-02-10T05:57:32.855840+0000","last_became_peered":"2024-02-10T05:57:32.855840+0000","last_unstale":"2024-02-10T05:58:14.901278+0000","last_undegraded":"2024-02-10T05:58:14.901278+0000","last_fullsized":"2024-02-10T05:58:14.901278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:17.586477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"70'16","reported_seq":56,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367215+0000","last_change":"2024-02-10T05:57:35.045180+0000","last_active":"2024-02-10T05:58:15.367215+0000","last_peered":"2024-02-10T05:58:15.367215+0000","last_clean":"2024-02-10T05:58:15.367215+0000","last_became_active":"2024-02-10T05:57:35.044871+0000","last_became_peered":"2024-02-10T05:57:35.044871+0000","last_unstale":"2024-02-10T05:58:15.367215+0000","last_undegraded":"2024-02-10T05:58:15.367215+0000","last_fullsized":"2024-02-10T05:58:15.367215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:46:14.429583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901868+0000","last_change":"2024-02-10T05:57:38.902944+0000","last_active":"2024-02-10T05:58:14.901868+0000","last_peered":"2024-02-10T05:58:14.901868+0000","last_clean":"2024-02-10T05:58:14.901868+0000","last_became_active":"2024-02-10T05:57:38.902602+0000","last_became_peered":"2024-02-10T05:57:38.902602+0000","last_unstale":"2024-02-10T05:58:14.901868+0000","last_undegraded":"2024-02-10T05:58:14.901868+0000","last_fullsized":"2024-02-10T05:58:14.901868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:30.802375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367833+0000","last_change":"2024-02-10T05:57:36.904991+0000","last_active":"2024-02-10T05:58:15.367833+0000","last_peered":"2024-02-10T05:58:15.367833+0000","last_clean":"2024-02-10T05:58:15.367833+0000","last_became_active":"2024-02-10T05:57:36.904654+0000","last_became_peered":"2024-02-10T05:57:36.904654+0000","last_unstale":"2024-02-10T05:58:15.367833+0000","last_undegraded":"2024-02-10T05:58:15.367833+0000","last_fullsized":"2024-02-10T05:58:15.367833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:58:30.565643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902504+0000","last_change":"2024-02-10T05:57:32.860854+0000","last_active":"2024-02-10T05:58:14.902504+0000","last_peered":"2024-02-10T05:58:14.902504+0000","last_clean":"2024-02-10T05:58:14.902504+0000","last_became_active":"2024-02-10T05:57:32.860536+0000","last_became_peered":"2024-02-10T05:57:32.860536+0000","last_unstale":"2024-02-10T05:58:14.902504+0000","last_undegraded":"2024-02-10T05:58:14.902504+0000","last_fullsized":"2024-02-10T05:58:14.902504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:18:32.882089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"70'17","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898497+0000","last_change":"2024-02-10T05:57:34.879340+0000","last_active":"2024-02-10T05:58:14.898497+0000","last_peered":"2024-02-10T05:58:14.898497+0000","last_clean":"2024-02-10T05:58:14.898497+0000","last_became_active":"2024-02-10T05:57:34.879049+0000","last_became_peered":"2024-02-10T05:57:34.879049+0000","last_unstale":"2024-02-10T05:58:14.898497+0000","last_undegraded":"2024-02-10T05:58:14.898497+0000","last_fullsized":"2024-02-10T05:58:14.898497+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:12:20.569625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016390+0000","last_change":"2024-02-10T05:57:38.908130+0000","last_active":"2024-02-10T05:58:15.016390+0000","last_peered":"2024-02-10T05:58:15.016390+0000","last_clean":"2024-02-10T05:58:15.016390+0000","last_became_active":"2024-02-10T05:57:38.907815+0000","last_became_peered":"2024-02-10T05:57:38.907815+0000","last_unstale":"2024-02-10T05:58:15.016390+0000","last_undegraded":"2024-02-10T05:58:15.016390+0000","last_fullsized":"2024-02-10T05:58:15.016390+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:40:53.616314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.897815+0000","last_change":"2024-02-10T05:57:36.902763+0000","last_active":"2024-02-10T05:58:14.897815+0000","last_peered":"2024-02-10T05:58:14.897815+0000","last_clean":"2024-02-10T05:58:14.897815+0000","last_became_active":"2024-02-10T05:57:36.902475+0000","last_became_peered":"2024-02-10T05:57:36.902475+0000","last_unstale":"2024-02-10T05:58:14.897815+0000","last_undegraded":"2024-02-10T05:58:14.897815+0000","last_fullsized":"2024-02-10T05:58:14.897815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:29:39.504947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368006+0000","last_change":"2024-02-10T05:57:32.862782+0000","last_active":"2024-02-10T05:58:15.368006+0000","last_peered":"2024-02-10T05:58:15.368006+0000","last_clean":"2024-02-10T05:58:15.368006+0000","last_became_active":"2024-02-10T05:57:32.862535+0000","last_became_peered":"2024-02-10T05:57:32.862535+0000","last_unstale":"2024-02-10T05:58:15.368006+0000","last_undegraded":"2024-02-10T05:58:15.368006+0000","last_fullsized":"2024-02-10T05:58:15.368006+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:52:24.286257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"70'4","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902604+0000","last_change":"2024-02-10T05:57:34.881284+0000","last_active":"2024-02-10T05:58:14.902604+0000","last_peered":"2024-02-10T05:58:14.902604+0000","last_clean":"2024-02-10T05:58:14.902604+0000","last_became_active":"2024-02-10T05:57:34.880992+0000","last_became_peered":"2024-02-10T05:57:34.880992+0000","last_unstale":"2024-02-10T05:58:14.902604+0000","last_undegraded":"2024-02-10T05:58:14.902604+0000","last_fullsized":"2024-02-10T05:58:14.902604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:29:24.596639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903166+0000","last_change":"2024-02-10T05:57:38.946553+0000","last_active":"2024-02-10T05:58:14.903166+0000","last_peered":"2024-02-10T05:58:14.903166+0000","last_clean":"2024-02-10T05:58:14.903166+0000","last_became_active":"2024-02-10T05:57:38.946246+0000","last_became_peered":"2024-02-10T05:57:38.946246+0000","last_unstale":"2024-02-10T05:58:14.903166+0000","last_undegraded":"2024-02-10T05:58:14.903166+0000","last_fullsized":"2024-02-10T05:58:14.903166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:52:58.904694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016622+0000","last_change":"2024-02-10T05:57:36.900143+0000","last_active":"2024-02-10T05:58:15.016622+0000","last_peered":"2024-02-10T05:58:15.016622+0000","last_clean":"2024-02-10T05:58:15.016622+0000","last_became_active":"2024-02-10T05:57:36.899880+0000","last_became_peered":"2024-02-10T05:57:36.899880+0000","last_unstale":"2024-02-10T05:58:15.016622+0000","last_undegraded":"2024-02-10T05:58:15.016622+0000","last_fullsized":"2024-02-10T05:58:15.016622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:31:13.020803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006495+0000","last_change":"2024-02-10T05:57:32.861901+0000","last_active":"2024-02-10T05:58:15.006495+0000","last_peered":"2024-02-10T05:58:15.006495+0000","last_clean":"2024-02-10T05:58:15.006495+0000","last_became_active":"2024-02-10T05:57:32.861634+0000","last_became_peered":"2024-02-10T05:57:32.861634+0000","last_unstale":"2024-02-10T05:58:15.006495+0000","last_undegraded":"2024-02-10T05:58:15.006495+0000","last_fullsized":"2024-02-10T05:58:15.006495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:29:55.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"70'19","reported_seq":62,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902987+0000","last_change":"2024-02-10T05:57:35.046407+0000","last_active":"2024-02-10T05:58:14.902987+0000","last_peered":"2024-02-10T05:58:14.902987+0000","last_clean":"2024-02-10T05:58:14.902987+0000","last_became_active":"2024-02-10T05:57:35.045948+0000","last_became_peered":"2024-02-10T05:57:35.045948+0000","last_unstale":"2024-02-10T05:58:14.902987+0000","last_undegraded":"2024-02-10T05:58:14.902987+0000","last_fullsized":"2024-02-10T05:58:14.902987+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:11:41.381051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367103+0000","last_change":"2024-02-10T05:57:38.909272+0000","last_active":"2024-02-10T05:58:15.367103+0000","last_peered":"2024-02-10T05:58:15.367103+0000","last_clean":"2024-02-10T05:58:15.367103+0000","last_became_active":"2024-02-10T05:57:38.908989+0000","last_became_peered":"2024-02-10T05:57:38.908989+0000","last_unstale":"2024-02-10T05:58:15.367103+0000","last_undegraded":"2024-02-10T05:58:15.367103+0000","last_fullsized":"2024-02-10T05:58:15.367103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:35:45.040983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903386+0000","last_change":"2024-02-10T05:57:36.900935+0000","last_active":"2024-02-10T05:58:14.903386+0000","last_peered":"2024-02-10T05:58:14.903386+0000","last_clean":"2024-02-10T05:58:14.903386+0000","last_became_active":"2024-02-10T05:57:36.900607+0000","last_became_peered":"2024-02-10T05:57:36.900607+0000","last_unstale":"2024-02-10T05:58:14.903386+0000","last_undegraded":"2024-02-10T05:58:14.903386+0000","last_fullsized":"2024-02-10T05:58:14.903386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:43:09.416948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016862+0000","last_change":"2024-02-10T05:57:32.858221+0000","last_active":"2024-02-10T05:58:15.016862+0000","last_peered":"2024-02-10T05:58:15.016862+0000","last_clean":"2024-02-10T05:58:15.016862+0000","last_became_active":"2024-02-10T05:57:32.857917+0000","last_became_peered":"2024-02-10T05:57:32.857917+0000","last_unstale":"2024-02-10T05:58:15.016862+0000","last_undegraded":"2024-02-10T05:58:15.016862+0000","last_fullsized":"2024-02-10T05:58:15.016862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:47:29.956245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"70'9","reported_seq":45,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901664+0000","last_change":"2024-02-10T05:57:34.878770+0000","last_active":"2024-02-10T05:58:14.901664+0000","last_peered":"2024-02-10T05:58:14.901664+0000","last_clean":"2024-02-10T05:58:14.901664+0000","last_became_active":"2024-02-10T05:57:34.877316+0000","last_became_peered":"2024-02-10T05:57:34.877316+0000","last_unstale":"2024-02-10T05:58:14.901664+0000","last_undegraded":"2024-02-10T05:58:14.901664+0000","last_fullsized":"2024-02-10T05:58:14.901664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:48.254365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902508+0000","last_change":"2024-02-10T05:57:38.914191+0000","last_active":"2024-02-10T05:58:14.902508+0000","last_peered":"2024-02-10T05:58:14.902508+0000","last_clean":"2024-02-10T05:58:14.902508+0000","last_became_active":"2024-02-10T05:57:38.913852+0000","last_became_peered":"2024-02-10T05:57:38.913852+0000","last_unstale":"2024-02-10T05:58:14.902508+0000","last_undegraded":"2024-02-10T05:58:14.902508+0000","last_fullsized":"2024-02-10T05:58:14.902508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:56:36.781275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901565+0000","last_change":"2024-02-10T05:57:36.905185+0000","last_active":"2024-02-10T05:58:14.901565+0000","last_peered":"2024-02-10T05:58:14.901565+0000","last_clean":"2024-02-10T05:58:14.901565+0000","last_became_active":"2024-02-10T05:57:36.904868+0000","last_became_peered":"2024-02-10T05:57:36.904868+0000","last_unstale":"2024-02-10T05:58:14.901565+0000","last_undegraded":"2024-02-10T05:58:14.901565+0000","last_fullsized":"2024-02-10T05:58:14.901565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:00:28.594435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049043+0000","last_change":"2024-02-10T05:57:32.862909+0000","last_active":"2024-02-10T05:58:15.049043+0000","last_peered":"2024-02-10T05:58:15.049043+0000","last_clean":"2024-02-10T05:58:15.049043+0000","last_became_active":"2024-02-10T05:57:32.862625+0000","last_became_peered":"2024-02-10T05:57:32.862625+0000","last_unstale":"2024-02-10T05:58:15.049043+0000","last_undegraded":"2024-02-10T05:58:15.049043+0000","last_fullsized":"2024-02-10T05:58:15.049043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:06.559322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"70'11","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048731+0000","last_change":"2024-02-10T05:57:53.974357+0000","last_active":"2024-02-10T05:58:15.048731+0000","last_peered":"2024-02-10T05:58:15.048731+0000","last_clean":"2024-02-10T05:58:15.048731+0000","last_became_active":"2024-02-10T05:57:53.968901+0000","last_became_peered":"2024-02-10T05:57:53.968901+0000","last_unstale":"2024-02-10T05:58:15.048731+0000","last_undegraded":"2024-02-10T05:58:15.048731+0000","last_fullsized":"2024-02-10T05:58:15.048731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:54.212906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367396+0000","last_change":"2024-02-10T05:57:38.913119+0000","last_active":"2024-02-10T05:58:15.367396+0000","last_peered":"2024-02-10T05:58:15.367396+0000","last_clean":"2024-02-10T05:58:15.367396+0000","last_became_active":"2024-02-10T05:57:38.912781+0000","last_became_peered":"2024-02-10T05:57:38.912781+0000","last_unstale":"2024-02-10T05:58:15.367396+0000","last_undegraded":"2024-02-10T05:58:15.367396+0000","last_fullsized":"2024-02-10T05:58:15.367396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:01:08.197833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"70'8","reported_seq":51,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198616+0000","last_change":"2024-02-10T05:57:36.906137+0000","last_active":"2024-02-10T05:58:38.198616+0000","last_peered":"2024-02-10T05:58:38.198616+0000","last_clean":"2024-02-10T05:58:38.198616+0000","last_became_active":"2024-02-10T05:57:36.905772+0000","last_became_peered":"2024-02-10T05:57:36.905772+0000","last_unstale":"2024-02-10T05:58:38.198616+0000","last_undegraded":"2024-02-10T05:58:38.198616+0000","last_fullsized":"2024-02-10T05:58:38.198616+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:38:14.267784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016494+0000","last_change":"2024-02-10T05:57:33.146307+0000","last_active":"2024-02-10T05:58:15.016494+0000","last_peered":"2024-02-10T05:58:15.016494+0000","last_clean":"2024-02-10T05:58:15.016494+0000","last_became_active":"2024-02-10T05:57:33.145780+0000","last_became_peered":"2024-02-10T05:57:33.145780+0000","last_unstale":"2024-02-10T05:58:15.016494+0000","last_undegraded":"2024-02-10T05:58:15.016494+0000","last_fullsized":"2024-02-10T05:58:15.016494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:52:16.417321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"70'22","reported_seq":68,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903067+0000","last_change":"2024-02-10T05:57:35.046480+0000","last_active":"2024-02-10T05:58:14.903067+0000","last_peered":"2024-02-10T05:58:14.903067+0000","last_clean":"2024-02-10T05:58:14.903067+0000","last_became_active":"2024-02-10T05:57:35.046036+0000","last_became_peered":"2024-02-10T05:57:35.046036+0000","last_unstale":"2024-02-10T05:58:14.903067+0000","last_undegraded":"2024-02-10T05:58:14.903067+0000","last_fullsized":"2024-02-10T05:58:14.903067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:45:07.469244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049328+0000","last_change":"2024-02-10T05:57:38.907765+0000","last_active":"2024-02-10T05:58:15.049328+0000","last_peered":"2024-02-10T05:58:15.049328+0000","last_clean":"2024-02-10T05:58:15.049328+0000","last_became_active":"2024-02-10T05:57:38.907480+0000","last_became_peered":"2024-02-10T05:57:38.907480+0000","last_unstale":"2024-02-10T05:58:15.049328+0000","last_undegraded":"2024-02-10T05:58:15.049328+0000","last_fullsized":"2024-02-10T05:58:15.049328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:50.553789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198618+0000","last_change":"2024-02-10T05:57:36.898829+0000","last_active":"2024-02-10T05:58:38.198618+0000","last_peered":"2024-02-10T05:58:38.198618+0000","last_clean":"2024-02-10T05:58:38.198618+0000","last_became_active":"2024-02-10T05:57:36.898519+0000","last_became_peered":"2024-02-10T05:57:36.898519+0000","last_unstale":"2024-02-10T05:58:38.198618+0000","last_undegraded":"2024-02-10T05:58:38.198618+0000","last_fullsized":"2024-02-10T05:58:38.198618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:15:35.706792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367494+0000","last_change":"2024-02-10T05:57:32.858446+0000","last_active":"2024-02-10T05:58:15.367494+0000","last_peered":"2024-02-10T05:58:15.367494+0000","last_clean":"2024-02-10T05:58:15.367494+0000","last_became_active":"2024-02-10T05:57:32.858149+0000","last_became_peered":"2024-02-10T05:57:32.858149+0000","last_unstale":"2024-02-10T05:58:15.367494+0000","last_undegraded":"2024-02-10T05:58:15.367494+0000","last_fullsized":"2024-02-10T05:58:15.367494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:22:54.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"70'15","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006384+0000","last_change":"2024-02-10T05:57:35.046228+0000","last_active":"2024-02-10T05:58:15.006384+0000","last_peered":"2024-02-10T05:58:15.006384+0000","last_clean":"2024-02-10T05:58:15.006384+0000","last_became_active":"2024-02-10T05:57:35.045913+0000","last_became_peered":"2024-02-10T05:57:35.045913+0000","last_unstale":"2024-02-10T05:58:15.006384+0000","last_undegraded":"2024-02-10T05:58:15.006384+0000","last_fullsized":"2024-02-10T05:58:15.006384+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:57:49.958705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013724+0000","last_change":"2024-02-10T05:57:38.906333+0000","last_active":"2024-02-10T05:58:15.013724+0000","last_peered":"2024-02-10T05:58:15.013724+0000","last_clean":"2024-02-10T05:58:15.013724+0000","last_became_active":"2024-02-10T05:57:38.905728+0000","last_became_peered":"2024-02-10T05:57:38.905728+0000","last_unstale":"2024-02-10T05:58:15.013724+0000","last_undegraded":"2024-02-10T05:58:15.013724+0000","last_fullsized":"2024-02-10T05:58:15.013724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:18:37.724316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902838+0000","last_change":"2024-02-10T05:57:36.899649+0000","last_active":"2024-02-10T05:58:14.902838+0000","last_peered":"2024-02-10T05:58:14.902838+0000","last_clean":"2024-02-10T05:58:14.902838+0000","last_became_active":"2024-02-10T05:57:36.899365+0000","last_became_peered":"2024-02-10T05:57:36.899365+0000","last_unstale":"2024-02-10T05:58:14.902838+0000","last_undegraded":"2024-02-10T05:58:14.902838+0000","last_fullsized":"2024-02-10T05:58:14.902838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:49:23.103644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016460+0000","last_change":"2024-02-10T05:57:32.856765+0000","last_active":"2024-02-10T05:58:15.016460+0000","last_peered":"2024-02-10T05:58:15.016460+0000","last_clean":"2024-02-10T05:58:15.016460+0000","last_became_active":"2024-02-10T05:57:32.856229+0000","last_became_peered":"2024-02-10T05:57:32.856229+0000","last_unstale":"2024-02-10T05:58:15.016460+0000","last_undegraded":"2024-02-10T05:58:15.016460+0000","last_fullsized":"2024-02-10T05:58:15.016460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:34.214983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"70'21","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367048+0000","last_change":"2024-02-10T05:57:35.045357+0000","last_active":"2024-02-10T05:58:15.367048+0000","last_peered":"2024-02-10T05:58:15.367048+0000","last_clean":"2024-02-10T05:58:15.367048+0000","last_became_active":"2024-02-10T05:57:35.044976+0000","last_became_peered":"2024-02-10T05:57:35.044976+0000","last_unstale":"2024-02-10T05:58:15.367048+0000","last_undegraded":"2024-02-10T05:58:15.367048+0000","last_fullsized":"2024-02-10T05:58:15.367048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:44.398663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016718+0000","last_change":"2024-02-10T05:57:38.947904+0000","last_active":"2024-02-10T05:58:15.016718+0000","last_peered":"2024-02-10T05:58:15.016718+0000","last_clean":"2024-02-10T05:58:15.016718+0000","last_became_active":"2024-02-10T05:57:38.947597+0000","last_became_peered":"2024-02-10T05:57:38.947597+0000","last_unstale":"2024-02-10T05:58:15.016718+0000","last_undegraded":"2024-02-10T05:58:15.016718+0000","last_fullsized":"2024-02-10T05:58:15.016718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:49.044062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006519+0000","last_change":"2024-02-10T05:57:36.904166+0000","last_active":"2024-02-10T05:58:15.006519+0000","last_peered":"2024-02-10T05:58:15.006519+0000","last_clean":"2024-02-10T05:58:15.006519+0000","last_became_active":"2024-02-10T05:57:36.903866+0000","last_became_peered":"2024-02-10T05:57:36.903866+0000","last_unstale":"2024-02-10T05:58:15.006519+0000","last_undegraded":"2024-02-10T05:58:15.006519+0000","last_fullsized":"2024-02-10T05:58:15.006519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:45.826388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.897762+0000","last_change":"2024-02-10T05:57:32.864171+0000","last_active":"2024-02-10T05:58:14.897762+0000","last_peered":"2024-02-10T05:58:14.897762+0000","last_clean":"2024-02-10T05:58:14.897762+0000","last_became_active":"2024-02-10T05:57:32.863648+0000","last_became_peered":"2024-02-10T05:57:32.863648+0000","last_unstale":"2024-02-10T05:58:14.897762+0000","last_undegraded":"2024-02-10T05:58:14.897762+0000","last_fullsized":"2024-02-10T05:58:14.897762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:36:33.391451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"70'12","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902460+0000","last_change":"2024-02-10T05:57:53.835070+0000","last_active":"2024-02-10T05:58:14.902460+0000","last_peered":"2024-02-10T05:58:14.902460+0000","last_clean":"2024-02-10T05:58:14.902460+0000","last_became_active":"2024-02-10T05:57:53.828730+0000","last_became_peered":"2024-02-10T05:57:53.828730+0000","last_unstale":"2024-02-10T05:58:14.902460+0000","last_undegraded":"2024-02-10T05:58:14.902460+0000","last_fullsized":"2024-02-10T05:58:14.902460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:05:02.109147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"70'1","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006147+0000","last_change":"2024-02-10T05:57:38.946775+0000","last_active":"2024-02-10T05:58:15.006147+0000","last_peered":"2024-02-10T05:58:15.006147+0000","last_clean":"2024-02-10T05:58:15.006147+0000","last_became_active":"2024-02-10T05:57:38.946496+0000","last_became_peered":"2024-02-10T05:57:38.946496+0000","last_unstale":"2024-02-10T05:58:15.006147+0000","last_undegraded":"2024-02-10T05:58:15.006147+0000","last_fullsized":"2024-02-10T05:58:15.006147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:37:04.535451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'2","reported_seq":45,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017139+0000","last_change":"2024-02-10T05:57:32.861323+0000","last_active":"2024-02-10T05:58:15.017139+0000","last_peered":"2024-02-10T05:58:15.017139+0000","last_clean":"2024-02-10T05:58:15.017139+0000","last_became_active":"2024-02-10T05:57:32.861045+0000","last_became_peered":"2024-02-10T05:57:32.861045+0000","last_unstale":"2024-02-10T05:58:15.017139+0000","last_undegraded":"2024-02-10T05:58:15.017139+0000","last_fullsized":"2024-02-10T05:58:15.017139+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:19:05.242990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"70'27","reported_seq":78,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005834+0000","last_change":"2024-02-10T05:57:35.046420+0000","last_active":"2024-02-10T05:58:15.005834+0000","last_peered":"2024-02-10T05:58:15.005834+0000","last_clean":"2024-02-10T05:58:15.005834+0000","last_became_active":"2024-02-10T05:57:35.045304+0000","last_became_peered":"2024-02-10T05:57:35.045304+0000","last_unstale":"2024-02-10T05:58:15.005834+0000","last_undegraded":"2024-02-10T05:58:15.005834+0000","last_fullsized":"2024-02-10T05:58:15.005834+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:25:12.720966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198618+0000","last_change":"2024-02-10T05:57:36.905987+0000","last_active":"2024-02-10T05:58:38.198618+0000","last_peered":"2024-02-10T05:58:38.198618+0000","last_clean":"2024-02-10T05:58:38.198618+0000","last_became_active":"2024-02-10T05:57:36.905600+0000","last_became_peered":"2024-02-10T05:57:36.905600+0000","last_unstale":"2024-02-10T05:58:38.198618+0000","last_undegraded":"2024-02-10T05:58:38.198618+0000","last_fullsized":"2024-02-10T05:58:38.198618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:00:15.291461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903277+0000","last_change":"2024-02-10T05:57:38.914363+0000","last_active":"2024-02-10T05:58:14.903277+0000","last_peered":"2024-02-10T05:58:14.903277+0000","last_clean":"2024-02-10T05:58:14.903277+0000","last_became_active":"2024-02-10T05:57:38.914022+0000","last_became_peered":"2024-02-10T05:57:38.914022+0000","last_unstale":"2024-02-10T05:58:14.903277+0000","last_undegraded":"2024-02-10T05:58:14.903277+0000","last_fullsized":"2024-02-10T05:58:14.903277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:22:19.635165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'1","reported_seq":39,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017050+0000","last_change":"2024-02-10T05:57:32.856574+0000","last_active":"2024-02-10T05:58:15.017050+0000","last_peered":"2024-02-10T05:58:15.017050+0000","last_clean":"2024-02-10T05:58:15.017050+0000","last_became_active":"2024-02-10T05:57:32.856284+0000","last_became_peered":"2024-02-10T05:57:32.856284+0000","last_unstale":"2024-02-10T05:58:15.017050+0000","last_undegraded":"2024-02-10T05:58:15.017050+0000","last_fullsized":"2024-02-10T05:58:15.017050+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:52:33.757963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"70'24","reported_seq":76,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048695+0000","last_change":"2024-02-10T05:57:34.882038+0000","last_active":"2024-02-10T05:58:15.048695+0000","last_peered":"2024-02-10T05:58:15.048695+0000","last_clean":"2024-02-10T05:58:15.048695+0000","last_became_active":"2024-02-10T05:57:34.881103+0000","last_became_peered":"2024-02-10T05:57:34.881103+0000","last_unstale":"2024-02-10T05:58:15.048695+0000","last_undegraded":"2024-02-10T05:58:15.048695+0000","last_fullsized":"2024-02-10T05:58:15.048695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:26:56.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013135+0000","last_change":"2024-02-10T05:57:36.902577+0000","last_active":"2024-02-10T05:58:15.013135+0000","last_peered":"2024-02-10T05:58:15.013135+0000","last_clean":"2024-02-10T05:58:15.013135+0000","last_became_active":"2024-02-10T05:57:36.902266+0000","last_became_peered":"2024-02-10T05:57:36.902266+0000","last_unstale":"2024-02-10T05:58:15.013135+0000","last_undegraded":"2024-02-10T05:58:15.013135+0000","last_fullsized":"2024-02-10T05:58:15.013135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:53:06.292543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016242+0000","last_change":"2024-02-10T05:57:38.910628+0000","last_active":"2024-02-10T05:58:15.016242+0000","last_peered":"2024-02-10T05:58:15.016242+0000","last_clean":"2024-02-10T05:58:15.016242+0000","last_became_active":"2024-02-10T05:57:38.910355+0000","last_became_peered":"2024-02-10T05:57:38.910355+0000","last_unstale":"2024-02-10T05:58:15.016242+0000","last_undegraded":"2024-02-10T05:58:15.016242+0000","last_fullsized":"2024-02-10T05:58:15.016242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:58:14.050942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013325+0000","last_change":"2024-02-10T05:57:36.903706+0000","last_active":"2024-02-10T05:58:15.013325+0000","last_peered":"2024-02-10T05:58:15.013325+0000","last_clean":"2024-02-10T05:58:15.013325+0000","last_became_active":"2024-02-10T05:57:36.903387+0000","last_became_peered":"2024-02-10T05:57:36.903387+0000","last_unstale":"2024-02-10T05:58:15.013325+0000","last_undegraded":"2024-02-10T05:58:15.013325+0000","last_fullsized":"2024-02-10T05:58:15.013325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:36:42.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016986+0000","last_change":"2024-02-10T05:57:32.860816+0000","last_active":"2024-02-10T05:58:15.016986+0000","last_peered":"2024-02-10T05:58:15.016986+0000","last_clean":"2024-02-10T05:58:15.016986+0000","last_became_active":"2024-02-10T05:57:32.860493+0000","last_became_peered":"2024-02-10T05:57:32.860493+0000","last_unstale":"2024-02-10T05:58:15.016986+0000","last_undegraded":"2024-02-10T05:58:15.016986+0000","last_fullsized":"2024-02-10T05:58:15.016986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:57:12.035724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"70'25","reported_seq":72,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898107+0000","last_change":"2024-02-10T05:57:35.045615+0000","last_active":"2024-02-10T05:58:14.898107+0000","last_peered":"2024-02-10T05:58:14.898107+0000","last_clean":"2024-02-10T05:58:14.898107+0000","last_became_active":"2024-02-10T05:57:35.045247+0000","last_became_peered":"2024-02-10T05:57:35.045247+0000","last_unstale":"2024-02-10T05:58:14.898107+0000","last_undegraded":"2024-02-10T05:58:14.898107+0000","last_fullsized":"2024-02-10T05:58:14.898107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:40:31.028585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367214+0000","last_change":"2024-02-10T05:57:38.907822+0000","last_active":"2024-02-10T05:58:15.367214+0000","last_peered":"2024-02-10T05:58:15.367214+0000","last_clean":"2024-02-10T05:58:15.367214+0000","last_became_active":"2024-02-10T05:57:38.907510+0000","last_became_peered":"2024-02-10T05:57:38.907510+0000","last_unstale":"2024-02-10T05:58:15.367214+0000","last_undegraded":"2024-02-10T05:58:15.367214+0000","last_fullsized":"2024-02-10T05:58:15.367214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:36:55.851283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013344+0000","last_change":"2024-02-10T05:57:36.894481+0000","last_active":"2024-02-10T05:58:15.013344+0000","last_peered":"2024-02-10T05:58:15.013344+0000","last_clean":"2024-02-10T05:58:15.013344+0000","last_became_active":"2024-02-10T05:57:36.894105+0000","last_became_peered":"2024-02-10T05:57:36.894105+0000","last_unstale":"2024-02-10T05:58:15.013344+0000","last_undegraded":"2024-02-10T05:58:15.013344+0000","last_fullsized":"2024-02-10T05:58:15.013344+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:16:35.789344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006244+0000","last_change":"2024-02-10T05:57:32.865967+0000","last_active":"2024-02-10T05:58:15.006244+0000","last_peered":"2024-02-10T05:58:15.006244+0000","last_clean":"2024-02-10T05:58:15.006244+0000","last_became_active":"2024-02-10T05:57:32.865656+0000","last_became_peered":"2024-02-10T05:57:32.865656+0000","last_unstale":"2024-02-10T05:58:15.006244+0000","last_undegraded":"2024-02-10T05:58:15.006244+0000","last_fullsized":"2024-02-10T05:58:15.006244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:45:07.315244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"70'15","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367970+0000","last_change":"2024-02-10T05:57:34.881314+0000","last_active":"2024-02-10T05:58:15.367970+0000","last_peered":"2024-02-10T05:58:15.367970+0000","last_clean":"2024-02-10T05:58:15.367970+0000","last_became_active":"2024-02-10T05:57:34.881024+0000","last_became_peered":"2024-02-10T05:57:34.881024+0000","last_unstale":"2024-02-10T05:58:15.367970+0000","last_undegraded":"2024-02-10T05:58:15.367970+0000","last_fullsized":"2024-02-10T05:58:15.367970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:24:55.650702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902669+0000","last_change":"2024-02-10T05:57:38.912952+0000","last_active":"2024-02-10T05:58:14.902669+0000","last_peered":"2024-02-10T05:58:14.902669+0000","last_clean":"2024-02-10T05:58:14.902669+0000","last_became_active":"2024-02-10T05:57:38.911996+0000","last_became_peered":"2024-02-10T05:57:38.911996+0000","last_unstale":"2024-02-10T05:58:14.902669+0000","last_undegraded":"2024-02-10T05:58:14.902669+0000","last_fullsized":"2024-02-10T05:58:14.902669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:23:25.541373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005971+0000","last_change":"2024-02-10T05:57:35.893626+0000","last_active":"2024-02-10T05:58:15.005971+0000","last_peered":"2024-02-10T05:58:15.005971+0000","last_clean":"2024-02-10T05:58:15.005971+0000","last_became_active":"2024-02-10T05:57:31.851364+0000","last_became_peered":"2024-02-10T05:57:31.851364+0000","last_unstale":"2024-02-10T05:58:15.005971+0000","last_undegraded":"2024-02-10T05:58:15.005971+0000","last_fullsized":"2024-02-10T05:58:15.005971+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:05:27.002023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020614230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048612+0000","last_change":"2024-02-10T05:57:32.865272+0000","last_active":"2024-02-10T05:58:15.048612+0000","last_peered":"2024-02-10T05:58:15.048612+0000","last_clean":"2024-02-10T05:58:15.048612+0000","last_became_active":"2024-02-10T05:57:32.864997+0000","last_became_peered":"2024-02-10T05:57:32.864997+0000","last_unstale":"2024-02-10T05:58:15.048612+0000","last_undegraded":"2024-02-10T05:58:15.048612+0000","last_fullsized":"2024-02-10T05:58:15.048612+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:29:36.850704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"70'34","reported_seq":104,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198427+0000","last_change":"2024-02-10T05:57:34.881001+0000","last_active":"2024-02-10T05:58:38.198427+0000","last_peered":"2024-02-10T05:58:38.198427+0000","last_clean":"2024-02-10T05:58:38.198427+0000","last_became_active":"2024-02-10T05:57:34.880701+0000","last_became_peered":"2024-02-10T05:57:34.880701+0000","last_unstale":"2024-02-10T05:58:38.198427+0000","last_undegraded":"2024-02-10T05:58:38.198427+0000","last_fullsized":"2024-02-10T05:58:38.198427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:00:32.434882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367602+0000","last_change":"2024-02-10T05:57:36.907335+0000","last_active":"2024-02-10T05:58:15.367602+0000","last_peered":"2024-02-10T05:58:15.367602+0000","last_clean":"2024-02-10T05:58:15.367602+0000","last_became_active":"2024-02-10T05:57:36.905411+0000","last_became_peered":"2024-02-10T05:57:36.905411+0000","last_unstale":"2024-02-10T05:58:15.367602+0000","last_undegraded":"2024-02-10T05:58:15.367602+0000","last_fullsized":"2024-02-10T05:58:15.367602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:21:32.554532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017088+0000","last_change":"2024-02-10T05:57:38.946373+0000","last_active":"2024-02-10T05:58:15.017088+0000","last_peered":"2024-02-10T05:58:15.017088+0000","last_clean":"2024-02-10T05:58:15.017088+0000","last_became_active":"2024-02-10T05:57:38.946084+0000","last_became_peered":"2024-02-10T05:57:38.946084+0000","last_unstale":"2024-02-10T05:58:15.017088+0000","last_undegraded":"2024-02-10T05:58:15.017088+0000","last_fullsized":"2024-02-10T05:58:15.017088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:25:52.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901215+0000","last_change":"2024-02-10T05:57:33.146153+0000","last_active":"2024-02-10T05:58:14.901215+0000","last_peered":"2024-02-10T05:58:14.901215+0000","last_clean":"2024-02-10T05:58:14.901215+0000","last_became_active":"2024-02-10T05:57:33.145791+0000","last_became_peered":"2024-02-10T05:57:33.145791+0000","last_unstale":"2024-02-10T05:58:14.901215+0000","last_undegraded":"2024-02-10T05:58:14.901215+0000","last_fullsized":"2024-02-10T05:58:14.901215+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:06:17.827042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013182+0000","last_change":"2024-02-10T05:57:35.886227+0000","last_active":"2024-02-10T05:58:15.013182+0000","last_peered":"2024-02-10T05:58:15.013182+0000","last_clean":"2024-02-10T05:58:15.013182+0000","last_became_active":"2024-02-10T05:57:31.855612+0000","last_became_peered":"2024-02-10T05:57:31.855612+0000","last_unstale":"2024-02-10T05:58:15.013182+0000","last_undegraded":"2024-02-10T05:58:15.013182+0000","last_fullsized":"2024-02-10T05:58:15.013182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:16:14.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044142500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"70'20","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902351+0000","last_change":"2024-02-10T05:57:34.878287+0000","last_active":"2024-02-10T05:58:14.902351+0000","last_peered":"2024-02-10T05:58:14.902351+0000","last_clean":"2024-02-10T05:58:14.902351+0000","last_became_active":"2024-02-10T05:57:34.877948+0000","last_became_peered":"2024-02-10T05:57:34.877948+0000","last_unstale":"2024-02-10T05:58:14.902351+0000","last_undegraded":"2024-02-10T05:58:14.902351+0000","last_fullsized":"2024-02-10T05:58:14.902351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:52.387764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013441+0000","last_change":"2024-02-10T05:57:36.901677+0000","last_active":"2024-02-10T05:58:15.013441+0000","last_peered":"2024-02-10T05:58:15.013441+0000","last_clean":"2024-02-10T05:58:15.013441+0000","last_became_active":"2024-02-10T05:57:36.901404+0000","last_became_peered":"2024-02-10T05:57:36.901404+0000","last_unstale":"2024-02-10T05:58:15.013441+0000","last_undegraded":"2024-02-10T05:58:15.013441+0000","last_fullsized":"2024-02-10T05:58:15.013441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:21:04.800761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006522+0000","last_change":"2024-02-10T05:57:38.910382+0000","last_active":"2024-02-10T05:58:15.006522+0000","last_peered":"2024-02-10T05:58:15.006522+0000","last_clean":"2024-02-10T05:58:15.006522+0000","last_became_active":"2024-02-10T05:57:38.910113+0000","last_became_peered":"2024-02-10T05:57:38.910113+0000","last_unstale":"2024-02-10T05:58:15.006522+0000","last_undegraded":"2024-02-10T05:58:15.006522+0000","last_fullsized":"2024-02-10T05:58:15.006522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:59:59.107005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'128","reported_seq":279,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:17.010754+0000","last_change":"2024-02-10T05:57:23.349116+0000","last_active":"2024-02-10T05:58:17.010754+0000","last_peered":"2024-02-10T05:58:17.010754+0000","last_clean":"2024-02-10T05:58:17.010754+0000","last_became_active":"2024-02-10T05:57:23.348655+0000","last_became_peered":"2024-02-10T05:57:23.348655+0000","last_unstale":"2024-02-10T05:58:17.010754+0000","last_undegraded":"2024-02-10T05:58:17.010754+0000","last_fullsized":"2024-02-10T05:58:17.010754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:13.055885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:13.055885+0000","last_clean_scrub_stamp":"2024-02-10T05:55:13.055885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:35:37.841146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902710+0000","last_change":"2024-02-10T05:57:32.857101+0000","last_active":"2024-02-10T05:58:14.902710+0000","last_peered":"2024-02-10T05:58:14.902710+0000","last_clean":"2024-02-10T05:58:14.902710+0000","last_became_active":"2024-02-10T05:57:32.856650+0000","last_became_peered":"2024-02-10T05:57:32.856650+0000","last_unstale":"2024-02-10T05:58:14.902710+0000","last_undegraded":"2024-02-10T05:58:14.902710+0000","last_fullsized":"2024-02-10T05:58:14.902710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:31:42.478191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"70'16","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198818+0000","last_change":"2024-02-10T05:57:35.045646+0000","last_active":"2024-02-10T05:58:38.198818+0000","last_peered":"2024-02-10T05:58:38.198818+0000","last_clean":"2024-02-10T05:58:38.198818+0000","last_became_active":"2024-02-10T05:57:35.045307+0000","last_became_peered":"2024-02-10T05:57:35.045307+0000","last_unstale":"2024-02-10T05:58:38.198818+0000","last_undegraded":"2024-02-10T05:58:38.198818+0000","last_fullsized":"2024-02-10T05:58:38.198818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:00:26.052403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016660+0000","last_change":"2024-02-10T05:57:36.905906+0000","last_active":"2024-02-10T05:58:15.016660+0000","last_peered":"2024-02-10T05:58:15.016660+0000","last_clean":"2024-02-10T05:58:15.016660+0000","last_became_active":"2024-02-10T05:57:36.905620+0000","last_became_peered":"2024-02-10T05:57:36.905620+0000","last_unstale":"2024-02-10T05:58:15.016660+0000","last_undegraded":"2024-02-10T05:58:15.016660+0000","last_fullsized":"2024-02-10T05:58:15.016660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:54:37.104479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901743+0000","last_change":"2024-02-10T05:57:38.910473+0000","last_active":"2024-02-10T05:58:14.901743+0000","last_peered":"2024-02-10T05:58:14.901743+0000","last_clean":"2024-02-10T05:58:14.901743+0000","last_became_active":"2024-02-10T05:57:38.910162+0000","last_became_peered":"2024-02-10T05:57:38.910162+0000","last_unstale":"2024-02-10T05:58:14.901743+0000","last_undegraded":"2024-02-10T05:58:14.901743+0000","last_fullsized":"2024-02-10T05:58:14.901743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:27:22.350693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367438+0000","last_change":"2024-02-10T05:57:32.859956+0000","last_active":"2024-02-10T05:58:15.367438+0000","last_peered":"2024-02-10T05:58:15.367438+0000","last_clean":"2024-02-10T05:58:15.367438+0000","last_became_active":"2024-02-10T05:57:32.859543+0000","last_became_peered":"2024-02-10T05:57:32.859543+0000","last_unstale":"2024-02-10T05:58:15.367438+0000","last_undegraded":"2024-02-10T05:58:15.367438+0000","last_fullsized":"2024-02-10T05:58:15.367438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:33.880345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"71'5","reported_seq":111,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:44.856853+0000","last_change":"2024-02-10T05:57:35.885973+0000","last_active":"2024-02-10T05:58:44.856853+0000","last_peered":"2024-02-10T05:58:44.856853+0000","last_clean":"2024-02-10T05:58:44.856853+0000","last_became_active":"2024-02-10T05:57:31.853279+0000","last_became_peered":"2024-02-10T05:57:31.853279+0000","last_unstale":"2024-02-10T05:58:44.856853+0000","last_undegraded":"2024-02-10T05:58:44.856853+0000","last_fullsized":"2024-02-10T05:58:44.856853+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:02:06.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371016,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":61,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"70'12","reported_seq":47,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367494+0000","last_change":"2024-02-10T05:57:34.882177+0000","last_active":"2024-02-10T05:58:15.367494+0000","last_peered":"2024-02-10T05:58:15.367494+0000","last_clean":"2024-02-10T05:58:15.367494+0000","last_became_active":"2024-02-10T05:57:34.881898+0000","last_became_peered":"2024-02-10T05:57:34.881898+0000","last_unstale":"2024-02-10T05:58:15.367494+0000","last_undegraded":"2024-02-10T05:58:15.367494+0000","last_fullsized":"2024-02-10T05:58:15.367494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:19:12.070557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006707+0000","last_change":"2024-02-10T05:57:36.906274+0000","last_active":"2024-02-10T05:58:15.006707+0000","last_peered":"2024-02-10T05:58:15.006707+0000","last_clean":"2024-02-10T05:58:15.006707+0000","last_became_active":"2024-02-10T05:57:36.905576+0000","last_became_peered":"2024-02-10T05:57:36.905576+0000","last_unstale":"2024-02-10T05:58:15.006707+0000","last_undegraded":"2024-02-10T05:58:15.006707+0000","last_fullsized":"2024-02-10T05:58:15.006707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:25:53.282213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901098+0000","last_change":"2024-02-10T05:57:38.946709+0000","last_active":"2024-02-10T05:58:14.901098+0000","last_peered":"2024-02-10T05:58:14.901098+0000","last_clean":"2024-02-10T05:58:14.901098+0000","last_became_active":"2024-02-10T05:57:38.946393+0000","last_became_peered":"2024-02-10T05:57:38.946393+0000","last_unstale":"2024-02-10T05:58:14.901098+0000","last_undegraded":"2024-02-10T05:58:14.901098+0000","last_fullsized":"2024-02-10T05:58:14.901098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:15:07.035953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901866+0000","last_change":"2024-02-10T05:57:33.146640+0000","last_active":"2024-02-10T05:58:14.901866+0000","last_peered":"2024-02-10T05:58:14.901866+0000","last_clean":"2024-02-10T05:58:14.901866+0000","last_became_active":"2024-02-10T05:57:33.146099+0000","last_became_peered":"2024-02-10T05:57:33.146099+0000","last_unstale":"2024-02-10T05:58:14.901866+0000","last_undegraded":"2024-02-10T05:58:14.901866+0000","last_fullsized":"2024-02-10T05:58:14.901866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:49.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"70'25","reported_seq":77,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367012+0000","last_change":"2024-02-10T05:57:35.045234+0000","last_active":"2024-02-10T05:58:15.367012+0000","last_peered":"2024-02-10T05:58:15.367012+0000","last_clean":"2024-02-10T05:58:15.367012+0000","last_became_active":"2024-02-10T05:57:35.044942+0000","last_became_peered":"2024-02-10T05:57:35.044942+0000","last_unstale":"2024-02-10T05:58:15.367012+0000","last_undegraded":"2024-02-10T05:58:15.367012+0000","last_fullsized":"2024-02-10T05:58:15.367012+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:55:45.304417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898231+0000","last_change":"2024-02-10T05:57:36.898973+0000","last_active":"2024-02-10T05:58:14.898231+0000","last_peered":"2024-02-10T05:58:14.898231+0000","last_clean":"2024-02-10T05:58:14.898231+0000","last_became_active":"2024-02-10T05:57:36.898676+0000","last_became_peered":"2024-02-10T05:57:36.898676+0000","last_unstale":"2024-02-10T05:58:14.898231+0000","last_undegraded":"2024-02-10T05:58:14.898231+0000","last_fullsized":"2024-02-10T05:58:14.898231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:40:40.205926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367126+0000","last_change":"2024-02-10T05:57:38.911648+0000","last_active":"2024-02-10T05:58:15.367126+0000","last_peered":"2024-02-10T05:58:15.367126+0000","last_clean":"2024-02-10T05:58:15.367126+0000","last_became_active":"2024-02-10T05:57:38.911369+0000","last_became_peered":"2024-02-10T05:57:38.911369+0000","last_unstale":"2024-02-10T05:58:15.367126+0000","last_undegraded":"2024-02-10T05:58:15.367126+0000","last_fullsized":"2024-02-10T05:58:15.367126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:03:59.716478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"63'1","reported_seq":39,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006317+0000","last_change":"2024-02-10T05:57:33.145375+0000","last_active":"2024-02-10T05:58:15.006317+0000","last_peered":"2024-02-10T05:58:15.006317+0000","last_clean":"2024-02-10T05:58:15.006317+0000","last_became_active":"2024-02-10T05:57:33.145028+0000","last_became_peered":"2024-02-10T05:57:33.145028+0000","last_unstale":"2024-02-10T05:58:15.006317+0000","last_undegraded":"2024-02-10T05:58:15.006317+0000","last_fullsized":"2024-02-10T05:58:15.006317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:14:57.252610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"70'16","reported_seq":56,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901572+0000","last_change":"2024-02-10T05:57:34.879043+0000","last_active":"2024-02-10T05:58:14.901572+0000","last_peered":"2024-02-10T05:58:14.901572+0000","last_clean":"2024-02-10T05:58:14.901572+0000","last_became_active":"2024-02-10T05:57:34.878435+0000","last_became_peered":"2024-02-10T05:57:34.878435+0000","last_unstale":"2024-02-10T05:58:14.901572+0000","last_undegraded":"2024-02-10T05:58:14.901572+0000","last_fullsized":"2024-02-10T05:58:14.901572+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:49:49.966790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198427+0000","last_change":"2024-02-10T05:57:36.903103+0000","last_active":"2024-02-10T05:58:38.198427+0000","last_peered":"2024-02-10T05:58:38.198427+0000","last_clean":"2024-02-10T05:58:38.198427+0000","last_became_active":"2024-02-10T05:57:36.902748+0000","last_became_peered":"2024-02-10T05:57:36.902748+0000","last_unstale":"2024-02-10T05:58:38.198427+0000","last_undegraded":"2024-02-10T05:58:38.198427+0000","last_fullsized":"2024-02-10T05:58:38.198427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:11:09.134559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016661+0000","last_change":"2024-02-10T05:57:38.948965+0000","last_active":"2024-02-10T05:58:15.016661+0000","last_peered":"2024-02-10T05:58:15.016661+0000","last_clean":"2024-02-10T05:58:15.016661+0000","last_became_active":"2024-02-10T05:57:38.948697+0000","last_became_peered":"2024-02-10T05:57:38.948697+0000","last_unstale":"2024-02-10T05:58:15.016661+0000","last_undegraded":"2024-02-10T05:58:15.016661+0000","last_fullsized":"2024-02-10T05:58:15.016661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:30.600684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367625+0000","last_change":"2024-02-10T05:57:32.858246+0000","last_active":"2024-02-10T05:58:15.367625+0000","last_peered":"2024-02-10T05:58:15.367625+0000","last_clean":"2024-02-10T05:58:15.367625+0000","last_became_active":"2024-02-10T05:57:32.857823+0000","last_became_peered":"2024-02-10T05:57:32.857823+0000","last_unstale":"2024-02-10T05:58:15.367625+0000","last_undegraded":"2024-02-10T05:58:15.367625+0000","last_fullsized":"2024-02-10T05:58:15.367625+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:02:28.187420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"70'22","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048622+0000","last_change":"2024-02-10T05:57:34.872173+0000","last_active":"2024-02-10T05:58:15.048622+0000","last_peered":"2024-02-10T05:58:15.048622+0000","last_clean":"2024-02-10T05:58:15.048622+0000","last_became_active":"2024-02-10T05:57:34.871826+0000","last_became_peered":"2024-02-10T05:57:34.871826+0000","last_unstale":"2024-02-10T05:58:15.048622+0000","last_undegraded":"2024-02-10T05:58:15.048622+0000","last_fullsized":"2024-02-10T05:58:15.048622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:31:14.669348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198503+0000","last_change":"2024-02-10T05:57:36.901651+0000","last_active":"2024-02-10T05:58:38.198503+0000","last_peered":"2024-02-10T05:58:38.198503+0000","last_clean":"2024-02-10T05:58:38.198503+0000","last_became_active":"2024-02-10T05:57:36.901368+0000","last_became_peered":"2024-02-10T05:57:36.901368+0000","last_unstale":"2024-02-10T05:58:38.198503+0000","last_undegraded":"2024-02-10T05:58:38.198503+0000","last_fullsized":"2024-02-10T05:58:38.198503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:02.531853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048522+0000","last_change":"2024-02-10T05:57:38.909800+0000","last_active":"2024-02-10T05:58:15.048522+0000","last_peered":"2024-02-10T05:58:15.048522+0000","last_clean":"2024-02-10T05:58:15.048522+0000","last_became_active":"2024-02-10T05:57:38.909491+0000","last_became_peered":"2024-02-10T05:57:38.909491+0000","last_unstale":"2024-02-10T05:58:15.048522+0000","last_undegraded":"2024-02-10T05:58:15.048522+0000","last_fullsized":"2024-02-10T05:58:15.048522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:04:57.916809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902309+0000","last_change":"2024-02-10T05:57:32.861540+0000","last_active":"2024-02-10T05:58:14.902309+0000","last_peered":"2024-02-10T05:58:14.902309+0000","last_clean":"2024-02-10T05:58:14.902309+0000","last_became_active":"2024-02-10T05:57:32.861130+0000","last_became_peered":"2024-02-10T05:57:32.861130+0000","last_unstale":"2024-02-10T05:58:14.902309+0000","last_undegraded":"2024-02-10T05:58:14.902309+0000","last_fullsized":"2024-02-10T05:58:14.902309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:45:27.247898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"70'30","reported_seq":85,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902791+0000","last_change":"2024-02-10T05:57:35.043917+0000","last_active":"2024-02-10T05:58:14.902791+0000","last_peered":"2024-02-10T05:58:14.902791+0000","last_clean":"2024-02-10T05:58:14.902791+0000","last_became_active":"2024-02-10T05:57:35.043616+0000","last_became_peered":"2024-02-10T05:57:35.043616+0000","last_unstale":"2024-02-10T05:58:14.902791+0000","last_undegraded":"2024-02-10T05:58:14.902791+0000","last_fullsized":"2024-02-10T05:58:14.902791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:15:31.903676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048435+0000","last_change":"2024-02-10T05:57:36.901508+0000","last_active":"2024-02-10T05:58:15.048435+0000","last_peered":"2024-02-10T05:58:15.048435+0000","last_clean":"2024-02-10T05:58:15.048435+0000","last_became_active":"2024-02-10T05:57:36.901249+0000","last_became_peered":"2024-02-10T05:57:36.901249+0000","last_unstale":"2024-02-10T05:58:15.048435+0000","last_undegraded":"2024-02-10T05:58:15.048435+0000","last_fullsized":"2024-02-10T05:58:15.048435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:51:38.104171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006121+0000","last_change":"2024-02-10T05:57:38.903424+0000","last_active":"2024-02-10T05:58:15.006121+0000","last_peered":"2024-02-10T05:58:15.006121+0000","last_clean":"2024-02-10T05:58:15.006121+0000","last_became_active":"2024-02-10T05:57:38.903197+0000","last_became_peered":"2024-02-10T05:57:38.903197+0000","last_unstale":"2024-02-10T05:58:15.006121+0000","last_undegraded":"2024-02-10T05:58:15.006121+0000","last_fullsized":"2024-02-10T05:58:15.006121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:27.767687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198621+0000","last_change":"2024-02-10T05:57:36.900643+0000","last_active":"2024-02-10T05:58:38.198621+0000","last_peered":"2024-02-10T05:58:38.198621+0000","last_clean":"2024-02-10T05:58:38.198621+0000","last_became_active":"2024-02-10T05:57:36.900358+0000","last_became_peered":"2024-02-10T05:57:36.900358+0000","last_unstale":"2024-02-10T05:58:38.198621+0000","last_undegraded":"2024-02-10T05:58:38.198621+0000","last_fullsized":"2024-02-10T05:58:38.198621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:08:53.572780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902707+0000","last_change":"2024-02-10T05:57:32.854246+0000","last_active":"2024-02-10T05:58:14.902707+0000","last_peered":"2024-02-10T05:58:14.902707+0000","last_clean":"2024-02-10T05:58:14.902707+0000","last_became_active":"2024-02-10T05:57:32.853756+0000","last_became_peered":"2024-02-10T05:57:32.853756+0000","last_unstale":"2024-02-10T05:58:14.902707+0000","last_undegraded":"2024-02-10T05:58:14.902707+0000","last_fullsized":"2024-02-10T05:58:14.902707+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:26:34.424373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"70'15","reported_seq":29,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013669+0000","last_change":"2024-02-10T05:57:53.837815+0000","last_active":"2024-02-10T05:58:15.013669+0000","last_peered":"2024-02-10T05:58:15.013669+0000","last_clean":"2024-02-10T05:58:15.013669+0000","last_became_active":"2024-02-10T05:57:53.826737+0000","last_became_peered":"2024-02-10T05:57:53.826737+0000","last_unstale":"2024-02-10T05:58:15.013669+0000","last_undegraded":"2024-02-10T05:58:15.013669+0000","last_fullsized":"2024-02-10T05:58:15.013669+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:38:29.273501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903039+0000","last_change":"2024-02-10T05:57:38.947474+0000","last_active":"2024-02-10T05:58:14.903039+0000","last_peered":"2024-02-10T05:58:14.903039+0000","last_clean":"2024-02-10T05:58:14.903039+0000","last_became_active":"2024-02-10T05:57:38.947048+0000","last_became_peered":"2024-02-10T05:57:38.947048+0000","last_unstale":"2024-02-10T05:58:14.903039+0000","last_undegraded":"2024-02-10T05:58:14.903039+0000","last_fullsized":"2024-02-10T05:58:14.903039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:34:06.693830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006656+0000","last_change":"2024-02-10T05:57:36.899334+0000","last_active":"2024-02-10T05:58:15.006656+0000","last_peered":"2024-02-10T05:58:15.006656+0000","last_clean":"2024-02-10T05:58:15.006656+0000","last_became_active":"2024-02-10T05:57:36.898922+0000","last_became_peered":"2024-02-10T05:57:36.898922+0000","last_unstale":"2024-02-10T05:58:15.006656+0000","last_undegraded":"2024-02-10T05:58:15.006656+0000","last_fullsized":"2024-02-10T05:58:15.006656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:02.111313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048501+0000","last_change":"2024-02-10T05:57:33.145437+0000","last_active":"2024-02-10T05:58:15.048501+0000","last_peered":"2024-02-10T05:58:15.048501+0000","last_clean":"2024-02-10T05:58:15.048501+0000","last_became_active":"2024-02-10T05:57:33.145143+0000","last_became_peered":"2024-02-10T05:57:33.145143+0000","last_unstale":"2024-02-10T05:58:15.048501+0000","last_undegraded":"2024-02-10T05:58:15.048501+0000","last_fullsized":"2024-02-10T05:58:15.048501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:25:55.824341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"70'17","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049060+0000","last_change":"2024-02-10T05:57:34.881918+0000","last_active":"2024-02-10T05:58:15.049060+0000","last_peered":"2024-02-10T05:58:15.049060+0000","last_clean":"2024-02-10T05:58:15.049060+0000","last_became_active":"2024-02-10T05:57:34.880894+0000","last_became_peered":"2024-02-10T05:57:34.880894+0000","last_unstale":"2024-02-10T05:58:15.049060+0000","last_undegraded":"2024-02-10T05:58:15.049060+0000","last_fullsized":"2024-02-10T05:58:15.049060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:31:25.928060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013810+0000","last_change":"2024-02-10T05:57:38.910919+0000","last_active":"2024-02-10T05:58:15.013810+0000","last_peered":"2024-02-10T05:58:15.013810+0000","last_clean":"2024-02-10T05:58:15.013810+0000","last_became_active":"2024-02-10T05:57:38.910602+0000","last_became_peered":"2024-02-10T05:57:38.910602+0000","last_unstale":"2024-02-10T05:58:15.013810+0000","last_undegraded":"2024-02-10T05:58:15.013810+0000","last_fullsized":"2024-02-10T05:58:15.013810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:27:27.004207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902175+0000","last_change":"2024-02-10T05:57:36.901504+0000","last_active":"2024-02-10T05:58:14.902175+0000","last_peered":"2024-02-10T05:58:14.902175+0000","last_clean":"2024-02-10T05:58:14.902175+0000","last_became_active":"2024-02-10T05:57:36.901215+0000","last_became_peered":"2024-02-10T05:57:36.901215+0000","last_unstale":"2024-02-10T05:58:14.902175+0000","last_undegraded":"2024-02-10T05:58:14.902175+0000","last_fullsized":"2024-02-10T05:58:14.902175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:00:28.028821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898131+0000","last_change":"2024-02-10T05:57:32.860729+0000","last_active":"2024-02-10T05:58:14.898131+0000","last_peered":"2024-02-10T05:58:14.898131+0000","last_clean":"2024-02-10T05:58:14.898131+0000","last_became_active":"2024-02-10T05:57:32.860426+0000","last_became_peered":"2024-02-10T05:57:32.860426+0000","last_unstale":"2024-02-10T05:58:14.898131+0000","last_undegraded":"2024-02-10T05:58:14.898131+0000","last_fullsized":"2024-02-10T05:58:14.898131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:48:05.831815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"70'17","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049265+0000","last_change":"2024-02-10T05:57:34.881319+0000","last_active":"2024-02-10T05:58:15.049265+0000","last_peered":"2024-02-10T05:58:15.049265+0000","last_clean":"2024-02-10T05:58:15.049265+0000","last_became_active":"2024-02-10T05:57:34.880995+0000","last_became_peered":"2024-02-10T05:57:34.880995+0000","last_unstale":"2024-02-10T05:58:15.049265+0000","last_undegraded":"2024-02-10T05:58:15.049265+0000","last_fullsized":"2024-02-10T05:58:15.049265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:45:40.948863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049227+0000","last_change":"2024-02-10T05:57:38.904960+0000","last_active":"2024-02-10T05:58:15.049227+0000","last_peered":"2024-02-10T05:58:15.049227+0000","last_clean":"2024-02-10T05:58:15.049227+0000","last_became_active":"2024-02-10T05:57:38.904757+0000","last_became_peered":"2024-02-10T05:57:38.904757+0000","last_unstale":"2024-02-10T05:58:15.049227+0000","last_undegraded":"2024-02-10T05:58:15.049227+0000","last_fullsized":"2024-02-10T05:58:15.049227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:36:30.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"70'8","reported_seq":48,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:38.198427+0000","last_change":"2024-02-10T05:57:36.900932+0000","last_active":"2024-02-10T05:58:38.198427+0000","last_peered":"2024-02-10T05:58:38.198427+0000","last_clean":"2024-02-10T05:58:38.198427+0000","last_became_active":"2024-02-10T05:57:36.900500+0000","last_became_peered":"2024-02-10T05:57:36.900500+0000","last_unstale":"2024-02-10T05:58:38.198427+0000","last_undegraded":"2024-02-10T05:58:38.198427+0000","last_fullsized":"2024-02-10T05:58:38.198427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:15.439510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903398+0000","last_change":"2024-02-10T05:57:32.860898+0000","last_active":"2024-02-10T05:58:14.903398+0000","last_peered":"2024-02-10T05:58:14.903398+0000","last_clean":"2024-02-10T05:58:14.903398+0000","last_became_active":"2024-02-10T05:57:32.860558+0000","last_became_peered":"2024-02-10T05:57:32.860558+0000","last_unstale":"2024-02-10T05:58:14.903398+0000","last_undegraded":"2024-02-10T05:58:14.903398+0000","last_fullsized":"2024-02-10T05:58:14.903398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:20:10.382846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"70'10","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048899+0000","last_change":"2024-02-10T05:57:34.874742+0000","last_active":"2024-02-10T05:58:15.048899+0000","last_peered":"2024-02-10T05:58:15.048899+0000","last_clean":"2024-02-10T05:58:15.048899+0000","last_became_active":"2024-02-10T05:57:34.874379+0000","last_became_peered":"2024-02-10T05:57:34.874379+0000","last_unstale":"2024-02-10T05:58:15.048899+0000","last_undegraded":"2024-02-10T05:58:15.048899+0000","last_fullsized":"2024-02-10T05:58:15.048899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:28.431131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":591,"ondisk_log_size":591,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":61,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168596,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32568,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712584,"statfs":{"total":95995035648,"available":95961686016,"internally_reserved":0,"allocated":1425408,"data_stored":937454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.54000000000000004}]},{"osd":4,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.891,"5min":5.891,"15min":5.891},"last":0.37},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46899999999999997}]},{"osd":6,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.42899999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1404928,"data_stored":936018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54200000000000004}]},{"osd":3,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52200000000000002}]},{"osd":4,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40799999999999997}]},{"osd":5,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.56899999999999995}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.66900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":347600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35099999999999998,"15min":0.32800000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54500000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.34000000000000002},"min":{"1min":0.28599999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.317}]},{"osd":3,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46999999999999997,"15min":0.48099999999999998},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.373},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.438,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Feb 10 05:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55500000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73199999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Feb 10 05:58:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.48799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82099999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80000000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1429504,"data_stored":936812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.35799999999999998,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.37,"15min":0.34999999999999998},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.66900000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35599999999999998}]},{"osd":2,"last update":"Sat Feb 10 05:58:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.47199999999999998,"15min":0.432},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.7009999999999996,"5min":4.7009999999999996,"15min":4.7009999999999996},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.47199999999999998,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Feb 10 05:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.50700000000000001,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.8019999999999996,"5min":4.8019999999999996,"15min":4.8019999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.5,"15min":0.46600000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.9699999999999998,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":0.36699999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:58:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.70599999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":1.377,"15min":1.377},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Feb 10 05:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.64100000000000001,"15min":0.61299999999999999},"min":{"1min":0.45100000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.6699999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.65200000000000002,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Feb 10 05:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":4.8200000000000003,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":4.9880000000000004,"5min":4.9880000000000004,"15min":4.9880000000000004},"last":0.56999999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":346735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.27000000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Feb 10 05:58:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51300000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.42899999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Feb 10 05:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67100000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.81899999999999995}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.80500000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":847872,"data_stored":347402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.253,"5min":0.184,"15min":0.184},"max":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.42399999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47099999999999997,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.245},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51500000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29199999999999998}]},{"osd":2,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57199999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54500000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.32600000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:58:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64900000000000002,"15min":0.65600000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.4359999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69299999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Feb 10 05:58:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78100000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67700000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.63400000000000001}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.621}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":839680,"data_stored":346640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.65200000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.41099999999999998}]},{"osd":3,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.45000000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:58:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.40300000000000002},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.41499999999999998,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.60899999999999999}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.627}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.64400000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":823296,"data_stored":346959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63300000000000001,"5min":0.872,"15min":0.872},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71199999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68899999999999995}]},{"osd":1,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.70099999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.75900000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.39100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69599999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72999999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.746}]},{"osd":4,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42599999999999999,"15min":0.42199999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.625}]},{"osd":6,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51300000000000001}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:58:51.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:58:52.217 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:52 smithi121 ceph-mon[41707]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:52.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:52 smithi121 ceph-mon[30020]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:52.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:52 smithi146 ceph-mon[33795]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:52.715 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:54.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:54 smithi121 ceph-mon[30020]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:54 smithi121 ceph-mon[41707]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:54.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:54 smithi146 ceph-mon[33795]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:55.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:58:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:58:56.269 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:56 smithi121 ceph-mon[41707]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:56.269 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:56 smithi121 ceph-mon[30020]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:56.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:56 smithi146 ceph-mon[33795]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:58:56.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:58:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:58:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.516+0000 7feb303ad700 1 -- 172.21.15.121:0/154469600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28107d50 msgr2=0x7feb28108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.516+0000 7feb303ad700 1 --2- 172.21.15.121:0/154469600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28107d50 0x7feb28108130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feb240099c0 tx=0x7feb24009cd0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 -- 172.21.15.121:0/154469600 shutdown_connections 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 --2- 172.21.15.121:0/154469600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28102730 0x7feb2810eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 --2- 172.21.15.121:0/154469600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28107d50 0x7feb28108130 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 --2- 172.21.15.121:0/154469600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb281021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 -- 172.21.15.121:0/154469600 >> 172.21.15.121:0/154469600 conn(0x7feb28078e50 msgr2=0x7feb28079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 -- 172.21.15.121:0/154469600 shutdown_connections 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 -- 172.21.15.121:0/154469600 wait complete. 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 Processor -- start 2024-02-10T05:58:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.517+0000 7feb303ad700 1 -- start start 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 0x7feb28197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28107d50 0x7feb2819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2819c5f0 con 0x7feb28102730 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2819c760 con 0x7feb28107d50 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2819c920 con 0x7feb28101d70 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2d948700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 0x7feb28197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:43866/0 (socket says 172.21.15.121:43866) 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 -- 172.21.15.121:0/641804264 learned_addr learned my addr 172.21.15.121:0/641804264 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e94a700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28107d50 0x7feb2819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 -- 172.21.15.121:0/641804264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28107d50 msgr2=0x7feb2819bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:56.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28107d50 0x7feb2819bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 -- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 msgr2=0x7feb28197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 0x7feb28197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 -- 172.21.15.121:0/641804264 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18009520 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2d948700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 0x7feb28197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb2e149700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb24005910 tx=0x7feb2400ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2401d070 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb240095f0 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7feb24003b00 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2400fa10 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.518+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb281a4290 con 0x7feb28101d70 2024-02-10T05:58:56.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.519+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb280689e0 con 0x7feb28101d70 2024-02-10T05:58:56.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.520+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7feb24003680 con 0x7feb28101d70 2024-02-10T05:58:56.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.520+0000 7feb1f7fe700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 0x7feb1406ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:58:56.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.520+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7feb2408e6e0 con 0x7feb28101d70 2024-02-10T05:58:56.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.520+0000 7feb2d948700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 0x7feb1406ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:58:56.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.521+0000 7feb2d948700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 0x7feb1406ff80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb18009fd0 tx=0x7feb18009260 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:58:56.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.525+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feb24057b70 con 0x7feb28101d70 2024-02-10T05:58:56.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.727+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 --> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb281112a0 con 0x7feb1406dac0 2024-02-10T05:58:56.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.737+0000 7feb1f7fe700 1 -- 172.21.15.121:0/641804264 <== mgr.24455 v2:172.21.15.121:6800/1142815253 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+384306 (secure 0 0 0) 0x7feb281112a0 con 0x7feb1406dac0 2024-02-10T05:58:56.743 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:58:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 msgr2=0x7feb1406ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 0x7feb1406ff80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feb18009fd0 tx=0x7feb18009260 comp rx=0 tx=0).stop 2024-02-10T05:58:56.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 msgr2=0x7feb28197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb24005910 tx=0x7feb2400ba30 comp rx=0 tx=0).stop 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 shutdown_connections 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7feb1406dac0 0x7feb1406ff80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb28107d50 0x7feb2819bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb28101d70 0x7feb28197760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 --2- 172.21.15.121:0/641804264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb28102730 0x7feb28197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.741+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 >> 172.21.15.121:0/641804264 conn(0x7feb28078e50 msgr2=0x7feb28105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.742+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 shutdown_connections 2024-02-10T05:58:56.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:58:56.742+0000 7feb303ad700 1 -- 172.21.15.121:0/641804264 wait complete. 2024-02-10T05:58:56.748 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-02-10T05:58:57.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:57.499 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:57.499 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:58:57.918 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-02-10T05:58:54.918823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1293,"num_read_kb":1107,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":798,"ondisk_log_size":798,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236888,"kb_used_data":8252,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724328,"statfs":{"total":767960285184,"available":767717711872,"internally_reserved":0,"allocated":8450048,"data_stored":4545620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":234016307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002732"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048473+0000","last_change":"2024-02-10T05:57:38.900690+0000","last_active":"2024-02-10T05:58:15.048473+0000","last_peered":"2024-02-10T05:58:15.048473+0000","last_clean":"2024-02-10T05:58:15.048473+0000","last_became_active":"2024-02-10T05:57:38.900430+0000","last_became_peered":"2024-02-10T05:57:38.900430+0000","last_unstale":"2024-02-10T05:58:15.048473+0000","last_undegraded":"2024-02-10T05:58:15.048473+0000","last_fullsized":"2024-02-10T05:58:15.048473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:19.909192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901193+0000","last_change":"2024-02-10T05:57:36.907884+0000","last_active":"2024-02-10T05:58:14.901193+0000","last_peered":"2024-02-10T05:58:14.901193+0000","last_clean":"2024-02-10T05:58:14.901193+0000","last_became_active":"2024-02-10T05:57:36.907441+0000","last_became_peered":"2024-02-10T05:57:36.907441+0000","last_unstale":"2024-02-10T05:58:14.901193+0000","last_undegraded":"2024-02-10T05:58:14.901193+0000","last_fullsized":"2024-02-10T05:58:14.901193+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:12:27.847516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368017+0000","last_change":"2024-02-10T05:57:33.145239+0000","last_active":"2024-02-10T05:58:15.368017+0000","last_peered":"2024-02-10T05:58:15.368017+0000","last_clean":"2024-02-10T05:58:15.368017+0000","last_became_active":"2024-02-10T05:57:33.144907+0000","last_became_peered":"2024-02-10T05:57:33.144907+0000","last_unstale":"2024-02-10T05:58:15.368017+0000","last_undegraded":"2024-02-10T05:58:15.368017+0000","last_fullsized":"2024-02-10T05:58:15.368017+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:13.513400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"70'16","reported_seq":58,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048383+0000","last_change":"2024-02-10T05:57:34.881439+0000","last_active":"2024-02-10T05:58:15.048383+0000","last_peered":"2024-02-10T05:58:15.048383+0000","last_clean":"2024-02-10T05:58:15.048383+0000","last_became_active":"2024-02-10T05:57:34.881048+0000","last_became_peered":"2024-02-10T05:57:34.881048+0000","last_unstale":"2024-02-10T05:58:15.048383+0000","last_undegraded":"2024-02-10T05:58:15.048383+0000","last_fullsized":"2024-02-10T05:58:15.048383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:24:04.210294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903287+0000","last_change":"2024-02-10T05:57:38.949010+0000","last_active":"2024-02-10T05:58:14.903287+0000","last_peered":"2024-02-10T05:58:14.903287+0000","last_clean":"2024-02-10T05:58:14.903287+0000","last_became_active":"2024-02-10T05:57:38.948657+0000","last_became_peered":"2024-02-10T05:57:38.948657+0000","last_unstale":"2024-02-10T05:58:14.903287+0000","last_undegraded":"2024-02-10T05:58:14.903287+0000","last_fullsized":"2024-02-10T05:58:14.903287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:43:15.197929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048295+0000","last_change":"2024-02-10T05:57:36.902625+0000","last_active":"2024-02-10T05:58:15.048295+0000","last_peered":"2024-02-10T05:58:15.048295+0000","last_clean":"2024-02-10T05:58:15.048295+0000","last_became_active":"2024-02-10T05:57:36.902344+0000","last_became_peered":"2024-02-10T05:57:36.902344+0000","last_unstale":"2024-02-10T05:58:15.048295+0000","last_undegraded":"2024-02-10T05:58:15.048295+0000","last_fullsized":"2024-02-10T05:58:15.048295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:01.464104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902819+0000","last_change":"2024-02-10T05:57:33.145205+0000","last_active":"2024-02-10T05:58:14.902819+0000","last_peered":"2024-02-10T05:58:14.902819+0000","last_clean":"2024-02-10T05:58:14.902819+0000","last_became_active":"2024-02-10T05:57:33.144875+0000","last_became_peered":"2024-02-10T05:57:33.144875+0000","last_unstale":"2024-02-10T05:58:14.902819+0000","last_undegraded":"2024-02-10T05:58:14.902819+0000","last_fullsized":"2024-02-10T05:58:14.902819+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:51:17.689770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"70'39","reported_seq":102,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903063+0000","last_change":"2024-02-10T05:57:34.878160+0000","last_active":"2024-02-10T05:58:14.903063+0000","last_peered":"2024-02-10T05:58:14.903063+0000","last_clean":"2024-02-10T05:58:14.903063+0000","last_became_active":"2024-02-10T05:57:34.877851+0000","last_became_peered":"2024-02-10T05:57:34.877851+0000","last_unstale":"2024-02-10T05:58:14.903063+0000","last_undegraded":"2024-02-10T05:58:14.903063+0000","last_fullsized":"2024-02-10T05:58:14.903063+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:20:23.895171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367731+0000","last_change":"2024-02-10T05:57:38.914265+0000","last_active":"2024-02-10T05:58:15.367731+0000","last_peered":"2024-02-10T05:58:15.367731+0000","last_clean":"2024-02-10T05:58:15.367731+0000","last_became_active":"2024-02-10T05:57:38.913956+0000","last_became_peered":"2024-02-10T05:57:38.913956+0000","last_unstale":"2024-02-10T05:58:15.367731+0000","last_undegraded":"2024-02-10T05:58:15.367731+0000","last_fullsized":"2024-02-10T05:58:15.367731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:36:01.753839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006068+0000","last_change":"2024-02-10T05:57:36.903461+0000","last_active":"2024-02-10T05:58:15.006068+0000","last_peered":"2024-02-10T05:58:15.006068+0000","last_clean":"2024-02-10T05:58:15.006068+0000","last_became_active":"2024-02-10T05:57:36.903183+0000","last_became_peered":"2024-02-10T05:57:36.903183+0000","last_unstale":"2024-02-10T05:58:15.006068+0000","last_undegraded":"2024-02-10T05:58:15.006068+0000","last_fullsized":"2024-02-10T05:58:15.006068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:51:01.700375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006192+0000","last_change":"2024-02-10T05:57:32.859268+0000","last_active":"2024-02-10T05:58:15.006192+0000","last_peered":"2024-02-10T05:58:15.006192+0000","last_clean":"2024-02-10T05:58:15.006192+0000","last_became_active":"2024-02-10T05:57:32.858544+0000","last_became_peered":"2024-02-10T05:57:32.858544+0000","last_unstale":"2024-02-10T05:58:15.006192+0000","last_undegraded":"2024-02-10T05:58:15.006192+0000","last_fullsized":"2024-02-10T05:58:15.006192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:59:58.445433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"70'21","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048633+0000","last_change":"2024-02-10T05:57:34.877523+0000","last_active":"2024-02-10T05:58:15.048633+0000","last_peered":"2024-02-10T05:58:15.048633+0000","last_clean":"2024-02-10T05:58:15.048633+0000","last_became_active":"2024-02-10T05:57:34.877220+0000","last_became_peered":"2024-02-10T05:57:34.877220+0000","last_unstale":"2024-02-10T05:58:15.048633+0000","last_undegraded":"2024-02-10T05:58:15.048633+0000","last_fullsized":"2024-02-10T05:58:15.048633+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:19.468024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005819+0000","last_change":"2024-02-10T05:57:38.909792+0000","last_active":"2024-02-10T05:58:15.005819+0000","last_peered":"2024-02-10T05:58:15.005819+0000","last_clean":"2024-02-10T05:58:15.005819+0000","last_became_active":"2024-02-10T05:57:38.909424+0000","last_became_peered":"2024-02-10T05:57:38.909424+0000","last_unstale":"2024-02-10T05:58:15.005819+0000","last_undegraded":"2024-02-10T05:58:15.005819+0000","last_fullsized":"2024-02-10T05:58:15.005819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:12:23.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016394+0000","last_change":"2024-02-10T05:57:36.904444+0000","last_active":"2024-02-10T05:58:15.016394+0000","last_peered":"2024-02-10T05:58:15.016394+0000","last_clean":"2024-02-10T05:58:15.016394+0000","last_became_active":"2024-02-10T05:57:36.904176+0000","last_became_peered":"2024-02-10T05:57:36.904176+0000","last_unstale":"2024-02-10T05:58:15.016394+0000","last_undegraded":"2024-02-10T05:58:15.016394+0000","last_fullsized":"2024-02-10T05:58:15.016394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:28:39.123394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006222+0000","last_change":"2024-02-10T05:57:32.862310+0000","last_active":"2024-02-10T05:58:15.006222+0000","last_peered":"2024-02-10T05:58:15.006222+0000","last_clean":"2024-02-10T05:58:15.006222+0000","last_became_active":"2024-02-10T05:57:32.862010+0000","last_became_peered":"2024-02-10T05:57:32.862010+0000","last_unstale":"2024-02-10T05:58:15.006222+0000","last_undegraded":"2024-02-10T05:58:15.006222+0000","last_fullsized":"2024-02-10T05:58:15.006222+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:40:15.874141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"70'11","reported_seq":51,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048891+0000","last_change":"2024-02-10T05:57:34.881679+0000","last_active":"2024-02-10T05:58:15.048891+0000","last_peered":"2024-02-10T05:58:15.048891+0000","last_clean":"2024-02-10T05:58:15.048891+0000","last_became_active":"2024-02-10T05:57:34.881054+0000","last_became_peered":"2024-02-10T05:57:34.881054+0000","last_unstale":"2024-02-10T05:58:15.048891+0000","last_undegraded":"2024-02-10T05:58:15.048891+0000","last_fullsized":"2024-02-10T05:58:15.048891+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:51.796412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049100+0000","last_change":"2024-02-10T05:57:38.946264+0000","last_active":"2024-02-10T05:58:15.049100+0000","last_peered":"2024-02-10T05:58:15.049100+0000","last_clean":"2024-02-10T05:58:15.049100+0000","last_became_active":"2024-02-10T05:57:38.945957+0000","last_became_peered":"2024-02-10T05:57:38.945957+0000","last_unstale":"2024-02-10T05:58:15.049100+0000","last_undegraded":"2024-02-10T05:58:15.049100+0000","last_fullsized":"2024-02-10T05:58:15.049100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:10.197651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902204+0000","last_change":"2024-02-10T05:57:36.901640+0000","last_active":"2024-02-10T05:58:14.902204+0000","last_peered":"2024-02-10T05:58:14.902204+0000","last_clean":"2024-02-10T05:58:14.902204+0000","last_became_active":"2024-02-10T05:57:36.901355+0000","last_became_peered":"2024-02-10T05:57:36.901355+0000","last_unstale":"2024-02-10T05:58:14.902204+0000","last_undegraded":"2024-02-10T05:58:14.902204+0000","last_fullsized":"2024-02-10T05:58:14.902204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:17:48.044556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368037+0000","last_change":"2024-02-10T05:57:32.861127+0000","last_active":"2024-02-10T05:58:15.368037+0000","last_peered":"2024-02-10T05:58:15.368037+0000","last_clean":"2024-02-10T05:58:15.368037+0000","last_became_active":"2024-02-10T05:57:32.860843+0000","last_became_peered":"2024-02-10T05:57:32.860843+0000","last_unstale":"2024-02-10T05:58:15.368037+0000","last_undegraded":"2024-02-10T05:58:15.368037+0000","last_fullsized":"2024-02-10T05:58:15.368037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:20:16.100213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"70'15","reported_seq":54,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013340+0000","last_change":"2024-02-10T05:57:34.879233+0000","last_active":"2024-02-10T05:58:15.013340+0000","last_peered":"2024-02-10T05:58:15.013340+0000","last_clean":"2024-02-10T05:58:15.013340+0000","last_became_active":"2024-02-10T05:57:34.878896+0000","last_became_peered":"2024-02-10T05:57:34.878896+0000","last_unstale":"2024-02-10T05:58:15.013340+0000","last_undegraded":"2024-02-10T05:58:15.013340+0000","last_fullsized":"2024-02-10T05:58:15.013340+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:08.661149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902161+0000","last_change":"2024-02-10T05:57:38.946919+0000","last_active":"2024-02-10T05:58:14.902161+0000","last_peered":"2024-02-10T05:58:14.902161+0000","last_clean":"2024-02-10T05:58:14.902161+0000","last_became_active":"2024-02-10T05:57:38.946234+0000","last_became_peered":"2024-02-10T05:57:38.946234+0000","last_unstale":"2024-02-10T05:58:14.902161+0000","last_undegraded":"2024-02-10T05:58:14.902161+0000","last_fullsized":"2024-02-10T05:58:14.902161+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:17:02.563845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048868+0000","last_change":"2024-02-10T05:57:36.894935+0000","last_active":"2024-02-10T05:58:15.048868+0000","last_peered":"2024-02-10T05:58:15.048868+0000","last_clean":"2024-02-10T05:58:15.048868+0000","last_became_active":"2024-02-10T05:57:36.894624+0000","last_became_peered":"2024-02-10T05:57:36.894624+0000","last_unstale":"2024-02-10T05:58:15.048868+0000","last_undegraded":"2024-02-10T05:58:15.048868+0000","last_fullsized":"2024-02-10T05:58:15.048868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:21:36.311046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048673+0000","last_change":"2024-02-10T05:57:33.146228+0000","last_active":"2024-02-10T05:58:15.048673+0000","last_peered":"2024-02-10T05:58:15.048673+0000","last_clean":"2024-02-10T05:58:15.048673+0000","last_became_active":"2024-02-10T05:57:33.145911+0000","last_became_peered":"2024-02-10T05:57:33.145911+0000","last_unstale":"2024-02-10T05:58:15.048673+0000","last_undegraded":"2024-02-10T05:58:15.048673+0000","last_fullsized":"2024-02-10T05:58:15.048673+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:58:23.951660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"70'18","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902262+0000","last_change":"2024-02-10T05:57:34.885839+0000","last_active":"2024-02-10T05:58:14.902262+0000","last_peered":"2024-02-10T05:58:14.902262+0000","last_clean":"2024-02-10T05:58:14.902262+0000","last_became_active":"2024-02-10T05:57:34.885491+0000","last_became_peered":"2024-02-10T05:57:34.885491+0000","last_unstale":"2024-02-10T05:58:14.902262+0000","last_undegraded":"2024-02-10T05:58:14.902262+0000","last_fullsized":"2024-02-10T05:58:14.902262+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:11:54.147255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"70'1","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016633+0000","last_change":"2024-02-10T05:57:38.907926+0000","last_active":"2024-02-10T05:58:15.016633+0000","last_peered":"2024-02-10T05:58:15.016633+0000","last_clean":"2024-02-10T05:58:15.016633+0000","last_became_active":"2024-02-10T05:57:38.907637+0000","last_became_peered":"2024-02-10T05:57:38.907637+0000","last_unstale":"2024-02-10T05:58:15.016633+0000","last_undegraded":"2024-02-10T05:58:15.016633+0000","last_fullsized":"2024-02-10T05:58:15.016633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:07:49.741587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"70'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199770+0000","last_change":"2024-02-10T05:57:36.900385+0000","last_active":"2024-02-10T05:58:43.199770+0000","last_peered":"2024-02-10T05:58:43.199770+0000","last_clean":"2024-02-10T05:58:43.199770+0000","last_became_active":"2024-02-10T05:57:36.899506+0000","last_became_peered":"2024-02-10T05:57:36.899506+0000","last_unstale":"2024-02-10T05:58:43.199770+0000","last_undegraded":"2024-02-10T05:58:43.199770+0000","last_fullsized":"2024-02-10T05:58:43.199770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:27:39.093215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"63'1","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901278+0000","last_change":"2024-02-10T05:57:32.856157+0000","last_active":"2024-02-10T05:58:14.901278+0000","last_peered":"2024-02-10T05:58:14.901278+0000","last_clean":"2024-02-10T05:58:14.901278+0000","last_became_active":"2024-02-10T05:57:32.855840+0000","last_became_peered":"2024-02-10T05:57:32.855840+0000","last_unstale":"2024-02-10T05:58:14.901278+0000","last_undegraded":"2024-02-10T05:58:14.901278+0000","last_fullsized":"2024-02-10T05:58:14.901278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:17.586477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"70'16","reported_seq":56,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367215+0000","last_change":"2024-02-10T05:57:35.045180+0000","last_active":"2024-02-10T05:58:15.367215+0000","last_peered":"2024-02-10T05:58:15.367215+0000","last_clean":"2024-02-10T05:58:15.367215+0000","last_became_active":"2024-02-10T05:57:35.044871+0000","last_became_peered":"2024-02-10T05:57:35.044871+0000","last_unstale":"2024-02-10T05:58:15.367215+0000","last_undegraded":"2024-02-10T05:58:15.367215+0000","last_fullsized":"2024-02-10T05:58:15.367215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:46:14.429583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901868+0000","last_change":"2024-02-10T05:57:38.902944+0000","last_active":"2024-02-10T05:58:14.901868+0000","last_peered":"2024-02-10T05:58:14.901868+0000","last_clean":"2024-02-10T05:58:14.901868+0000","last_became_active":"2024-02-10T05:57:38.902602+0000","last_became_peered":"2024-02-10T05:57:38.902602+0000","last_unstale":"2024-02-10T05:58:14.901868+0000","last_undegraded":"2024-02-10T05:58:14.901868+0000","last_fullsized":"2024-02-10T05:58:14.901868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:30.802375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367833+0000","last_change":"2024-02-10T05:57:36.904991+0000","last_active":"2024-02-10T05:58:15.367833+0000","last_peered":"2024-02-10T05:58:15.367833+0000","last_clean":"2024-02-10T05:58:15.367833+0000","last_became_active":"2024-02-10T05:57:36.904654+0000","last_became_peered":"2024-02-10T05:57:36.904654+0000","last_unstale":"2024-02-10T05:58:15.367833+0000","last_undegraded":"2024-02-10T05:58:15.367833+0000","last_fullsized":"2024-02-10T05:58:15.367833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:58:30.565643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902504+0000","last_change":"2024-02-10T05:57:32.860854+0000","last_active":"2024-02-10T05:58:14.902504+0000","last_peered":"2024-02-10T05:58:14.902504+0000","last_clean":"2024-02-10T05:58:14.902504+0000","last_became_active":"2024-02-10T05:57:32.860536+0000","last_became_peered":"2024-02-10T05:57:32.860536+0000","last_unstale":"2024-02-10T05:58:14.902504+0000","last_undegraded":"2024-02-10T05:58:14.902504+0000","last_fullsized":"2024-02-10T05:58:14.902504+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:18:32.882089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"70'17","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898497+0000","last_change":"2024-02-10T05:57:34.879340+0000","last_active":"2024-02-10T05:58:14.898497+0000","last_peered":"2024-02-10T05:58:14.898497+0000","last_clean":"2024-02-10T05:58:14.898497+0000","last_became_active":"2024-02-10T05:57:34.879049+0000","last_became_peered":"2024-02-10T05:57:34.879049+0000","last_unstale":"2024-02-10T05:58:14.898497+0000","last_undegraded":"2024-02-10T05:58:14.898497+0000","last_fullsized":"2024-02-10T05:58:14.898497+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:12:20.569625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016390+0000","last_change":"2024-02-10T05:57:38.908130+0000","last_active":"2024-02-10T05:58:15.016390+0000","last_peered":"2024-02-10T05:58:15.016390+0000","last_clean":"2024-02-10T05:58:15.016390+0000","last_became_active":"2024-02-10T05:57:38.907815+0000","last_became_peered":"2024-02-10T05:57:38.907815+0000","last_unstale":"2024-02-10T05:58:15.016390+0000","last_undegraded":"2024-02-10T05:58:15.016390+0000","last_fullsized":"2024-02-10T05:58:15.016390+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:40:53.616314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.897815+0000","last_change":"2024-02-10T05:57:36.902763+0000","last_active":"2024-02-10T05:58:14.897815+0000","last_peered":"2024-02-10T05:58:14.897815+0000","last_clean":"2024-02-10T05:58:14.897815+0000","last_became_active":"2024-02-10T05:57:36.902475+0000","last_became_peered":"2024-02-10T05:57:36.902475+0000","last_unstale":"2024-02-10T05:58:14.897815+0000","last_undegraded":"2024-02-10T05:58:14.897815+0000","last_fullsized":"2024-02-10T05:58:14.897815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:29:39.504947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.368006+0000","last_change":"2024-02-10T05:57:32.862782+0000","last_active":"2024-02-10T05:58:15.368006+0000","last_peered":"2024-02-10T05:58:15.368006+0000","last_clean":"2024-02-10T05:58:15.368006+0000","last_became_active":"2024-02-10T05:57:32.862535+0000","last_became_peered":"2024-02-10T05:57:32.862535+0000","last_unstale":"2024-02-10T05:58:15.368006+0000","last_undegraded":"2024-02-10T05:58:15.368006+0000","last_fullsized":"2024-02-10T05:58:15.368006+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:52:24.286257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"70'4","reported_seq":35,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902604+0000","last_change":"2024-02-10T05:57:34.881284+0000","last_active":"2024-02-10T05:58:14.902604+0000","last_peered":"2024-02-10T05:58:14.902604+0000","last_clean":"2024-02-10T05:58:14.902604+0000","last_became_active":"2024-02-10T05:57:34.880992+0000","last_became_peered":"2024-02-10T05:57:34.880992+0000","last_unstale":"2024-02-10T05:58:14.902604+0000","last_undegraded":"2024-02-10T05:58:14.902604+0000","last_fullsized":"2024-02-10T05:58:14.902604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:29:24.596639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903166+0000","last_change":"2024-02-10T05:57:38.946553+0000","last_active":"2024-02-10T05:58:14.903166+0000","last_peered":"2024-02-10T05:58:14.903166+0000","last_clean":"2024-02-10T05:58:14.903166+0000","last_became_active":"2024-02-10T05:57:38.946246+0000","last_became_peered":"2024-02-10T05:57:38.946246+0000","last_unstale":"2024-02-10T05:58:14.903166+0000","last_undegraded":"2024-02-10T05:58:14.903166+0000","last_fullsized":"2024-02-10T05:58:14.903166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:52:58.904694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016622+0000","last_change":"2024-02-10T05:57:36.900143+0000","last_active":"2024-02-10T05:58:15.016622+0000","last_peered":"2024-02-10T05:58:15.016622+0000","last_clean":"2024-02-10T05:58:15.016622+0000","last_became_active":"2024-02-10T05:57:36.899880+0000","last_became_peered":"2024-02-10T05:57:36.899880+0000","last_unstale":"2024-02-10T05:58:15.016622+0000","last_undegraded":"2024-02-10T05:58:15.016622+0000","last_fullsized":"2024-02-10T05:58:15.016622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:31:13.020803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006495+0000","last_change":"2024-02-10T05:57:32.861901+0000","last_active":"2024-02-10T05:58:15.006495+0000","last_peered":"2024-02-10T05:58:15.006495+0000","last_clean":"2024-02-10T05:58:15.006495+0000","last_became_active":"2024-02-10T05:57:32.861634+0000","last_became_peered":"2024-02-10T05:57:32.861634+0000","last_unstale":"2024-02-10T05:58:15.006495+0000","last_undegraded":"2024-02-10T05:58:15.006495+0000","last_fullsized":"2024-02-10T05:58:15.006495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:29:55.222461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"70'19","reported_seq":62,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902987+0000","last_change":"2024-02-10T05:57:35.046407+0000","last_active":"2024-02-10T05:58:14.902987+0000","last_peered":"2024-02-10T05:58:14.902987+0000","last_clean":"2024-02-10T05:58:14.902987+0000","last_became_active":"2024-02-10T05:57:35.045948+0000","last_became_peered":"2024-02-10T05:57:35.045948+0000","last_unstale":"2024-02-10T05:58:14.902987+0000","last_undegraded":"2024-02-10T05:58:14.902987+0000","last_fullsized":"2024-02-10T05:58:14.902987+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:11:41.381051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367103+0000","last_change":"2024-02-10T05:57:38.909272+0000","last_active":"2024-02-10T05:58:15.367103+0000","last_peered":"2024-02-10T05:58:15.367103+0000","last_clean":"2024-02-10T05:58:15.367103+0000","last_became_active":"2024-02-10T05:57:38.908989+0000","last_became_peered":"2024-02-10T05:57:38.908989+0000","last_unstale":"2024-02-10T05:58:15.367103+0000","last_undegraded":"2024-02-10T05:58:15.367103+0000","last_fullsized":"2024-02-10T05:58:15.367103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:35:45.040983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903386+0000","last_change":"2024-02-10T05:57:36.900935+0000","last_active":"2024-02-10T05:58:14.903386+0000","last_peered":"2024-02-10T05:58:14.903386+0000","last_clean":"2024-02-10T05:58:14.903386+0000","last_became_active":"2024-02-10T05:57:36.900607+0000","last_became_peered":"2024-02-10T05:57:36.900607+0000","last_unstale":"2024-02-10T05:58:14.903386+0000","last_undegraded":"2024-02-10T05:58:14.903386+0000","last_fullsized":"2024-02-10T05:58:14.903386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:43:09.416948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016862+0000","last_change":"2024-02-10T05:57:32.858221+0000","last_active":"2024-02-10T05:58:15.016862+0000","last_peered":"2024-02-10T05:58:15.016862+0000","last_clean":"2024-02-10T05:58:15.016862+0000","last_became_active":"2024-02-10T05:57:32.857917+0000","last_became_peered":"2024-02-10T05:57:32.857917+0000","last_unstale":"2024-02-10T05:58:15.016862+0000","last_undegraded":"2024-02-10T05:58:15.016862+0000","last_fullsized":"2024-02-10T05:58:15.016862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:47:29.956245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"70'9","reported_seq":45,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901664+0000","last_change":"2024-02-10T05:57:34.878770+0000","last_active":"2024-02-10T05:58:14.901664+0000","last_peered":"2024-02-10T05:58:14.901664+0000","last_clean":"2024-02-10T05:58:14.901664+0000","last_became_active":"2024-02-10T05:57:34.877316+0000","last_became_peered":"2024-02-10T05:57:34.877316+0000","last_unstale":"2024-02-10T05:58:14.901664+0000","last_undegraded":"2024-02-10T05:58:14.901664+0000","last_fullsized":"2024-02-10T05:58:14.901664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:48.254365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902508+0000","last_change":"2024-02-10T05:57:38.914191+0000","last_active":"2024-02-10T05:58:14.902508+0000","last_peered":"2024-02-10T05:58:14.902508+0000","last_clean":"2024-02-10T05:58:14.902508+0000","last_became_active":"2024-02-10T05:57:38.913852+0000","last_became_peered":"2024-02-10T05:57:38.913852+0000","last_unstale":"2024-02-10T05:58:14.902508+0000","last_undegraded":"2024-02-10T05:58:14.902508+0000","last_fullsized":"2024-02-10T05:58:14.902508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:56:36.781275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901565+0000","last_change":"2024-02-10T05:57:36.905185+0000","last_active":"2024-02-10T05:58:14.901565+0000","last_peered":"2024-02-10T05:58:14.901565+0000","last_clean":"2024-02-10T05:58:14.901565+0000","last_became_active":"2024-02-10T05:57:36.904868+0000","last_became_peered":"2024-02-10T05:57:36.904868+0000","last_unstale":"2024-02-10T05:58:14.901565+0000","last_undegraded":"2024-02-10T05:58:14.901565+0000","last_fullsized":"2024-02-10T05:58:14.901565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:00:28.594435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049043+0000","last_change":"2024-02-10T05:57:32.862909+0000","last_active":"2024-02-10T05:58:15.049043+0000","last_peered":"2024-02-10T05:58:15.049043+0000","last_clean":"2024-02-10T05:58:15.049043+0000","last_became_active":"2024-02-10T05:57:32.862625+0000","last_became_peered":"2024-02-10T05:57:32.862625+0000","last_unstale":"2024-02-10T05:58:15.049043+0000","last_undegraded":"2024-02-10T05:58:15.049043+0000","last_fullsized":"2024-02-10T05:58:15.049043+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:06.559322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"70'11","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048731+0000","last_change":"2024-02-10T05:57:53.974357+0000","last_active":"2024-02-10T05:58:15.048731+0000","last_peered":"2024-02-10T05:58:15.048731+0000","last_clean":"2024-02-10T05:58:15.048731+0000","last_became_active":"2024-02-10T05:57:53.968901+0000","last_became_peered":"2024-02-10T05:57:53.968901+0000","last_unstale":"2024-02-10T05:58:15.048731+0000","last_undegraded":"2024-02-10T05:58:15.048731+0000","last_fullsized":"2024-02-10T05:58:15.048731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:54.212906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367396+0000","last_change":"2024-02-10T05:57:38.913119+0000","last_active":"2024-02-10T05:58:15.367396+0000","last_peered":"2024-02-10T05:58:15.367396+0000","last_clean":"2024-02-10T05:58:15.367396+0000","last_became_active":"2024-02-10T05:57:38.912781+0000","last_became_peered":"2024-02-10T05:57:38.912781+0000","last_unstale":"2024-02-10T05:58:15.367396+0000","last_undegraded":"2024-02-10T05:58:15.367396+0000","last_fullsized":"2024-02-10T05:58:15.367396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:01:08.197833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"70'8","reported_seq":52,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199787+0000","last_change":"2024-02-10T05:57:36.906137+0000","last_active":"2024-02-10T05:58:43.199787+0000","last_peered":"2024-02-10T05:58:43.199787+0000","last_clean":"2024-02-10T05:58:43.199787+0000","last_became_active":"2024-02-10T05:57:36.905772+0000","last_became_peered":"2024-02-10T05:57:36.905772+0000","last_unstale":"2024-02-10T05:58:43.199787+0000","last_undegraded":"2024-02-10T05:58:43.199787+0000","last_fullsized":"2024-02-10T05:58:43.199787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:38:14.267784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016494+0000","last_change":"2024-02-10T05:57:33.146307+0000","last_active":"2024-02-10T05:58:15.016494+0000","last_peered":"2024-02-10T05:58:15.016494+0000","last_clean":"2024-02-10T05:58:15.016494+0000","last_became_active":"2024-02-10T05:57:33.145780+0000","last_became_peered":"2024-02-10T05:57:33.145780+0000","last_unstale":"2024-02-10T05:58:15.016494+0000","last_undegraded":"2024-02-10T05:58:15.016494+0000","last_fullsized":"2024-02-10T05:58:15.016494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:52:16.417321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"70'22","reported_seq":68,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903067+0000","last_change":"2024-02-10T05:57:35.046480+0000","last_active":"2024-02-10T05:58:14.903067+0000","last_peered":"2024-02-10T05:58:14.903067+0000","last_clean":"2024-02-10T05:58:14.903067+0000","last_became_active":"2024-02-10T05:57:35.046036+0000","last_became_peered":"2024-02-10T05:57:35.046036+0000","last_unstale":"2024-02-10T05:58:14.903067+0000","last_undegraded":"2024-02-10T05:58:14.903067+0000","last_fullsized":"2024-02-10T05:58:14.903067+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:45:07.469244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049328+0000","last_change":"2024-02-10T05:57:38.907765+0000","last_active":"2024-02-10T05:58:15.049328+0000","last_peered":"2024-02-10T05:58:15.049328+0000","last_clean":"2024-02-10T05:58:15.049328+0000","last_became_active":"2024-02-10T05:57:38.907480+0000","last_became_peered":"2024-02-10T05:57:38.907480+0000","last_unstale":"2024-02-10T05:58:15.049328+0000","last_undegraded":"2024-02-10T05:58:15.049328+0000","last_fullsized":"2024-02-10T05:58:15.049328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:50.553789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199637+0000","last_change":"2024-02-10T05:57:36.898829+0000","last_active":"2024-02-10T05:58:43.199637+0000","last_peered":"2024-02-10T05:58:43.199637+0000","last_clean":"2024-02-10T05:58:43.199637+0000","last_became_active":"2024-02-10T05:57:36.898519+0000","last_became_peered":"2024-02-10T05:57:36.898519+0000","last_unstale":"2024-02-10T05:58:43.199637+0000","last_undegraded":"2024-02-10T05:58:43.199637+0000","last_fullsized":"2024-02-10T05:58:43.199637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:15:35.706792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367494+0000","last_change":"2024-02-10T05:57:32.858446+0000","last_active":"2024-02-10T05:58:15.367494+0000","last_peered":"2024-02-10T05:58:15.367494+0000","last_clean":"2024-02-10T05:58:15.367494+0000","last_became_active":"2024-02-10T05:57:32.858149+0000","last_became_peered":"2024-02-10T05:57:32.858149+0000","last_unstale":"2024-02-10T05:58:15.367494+0000","last_undegraded":"2024-02-10T05:58:15.367494+0000","last_fullsized":"2024-02-10T05:58:15.367494+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:22:54.999834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"70'15","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006384+0000","last_change":"2024-02-10T05:57:35.046228+0000","last_active":"2024-02-10T05:58:15.006384+0000","last_peered":"2024-02-10T05:58:15.006384+0000","last_clean":"2024-02-10T05:58:15.006384+0000","last_became_active":"2024-02-10T05:57:35.045913+0000","last_became_peered":"2024-02-10T05:57:35.045913+0000","last_unstale":"2024-02-10T05:58:15.006384+0000","last_undegraded":"2024-02-10T05:58:15.006384+0000","last_fullsized":"2024-02-10T05:58:15.006384+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:57:49.958705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013724+0000","last_change":"2024-02-10T05:57:38.906333+0000","last_active":"2024-02-10T05:58:15.013724+0000","last_peered":"2024-02-10T05:58:15.013724+0000","last_clean":"2024-02-10T05:58:15.013724+0000","last_became_active":"2024-02-10T05:57:38.905728+0000","last_became_peered":"2024-02-10T05:57:38.905728+0000","last_unstale":"2024-02-10T05:58:15.013724+0000","last_undegraded":"2024-02-10T05:58:15.013724+0000","last_fullsized":"2024-02-10T05:58:15.013724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:18:37.724316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902838+0000","last_change":"2024-02-10T05:57:36.899649+0000","last_active":"2024-02-10T05:58:14.902838+0000","last_peered":"2024-02-10T05:58:14.902838+0000","last_clean":"2024-02-10T05:58:14.902838+0000","last_became_active":"2024-02-10T05:57:36.899365+0000","last_became_peered":"2024-02-10T05:57:36.899365+0000","last_unstale":"2024-02-10T05:58:14.902838+0000","last_undegraded":"2024-02-10T05:58:14.902838+0000","last_fullsized":"2024-02-10T05:58:14.902838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:49:23.103644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016460+0000","last_change":"2024-02-10T05:57:32.856765+0000","last_active":"2024-02-10T05:58:15.016460+0000","last_peered":"2024-02-10T05:58:15.016460+0000","last_clean":"2024-02-10T05:58:15.016460+0000","last_became_active":"2024-02-10T05:57:32.856229+0000","last_became_peered":"2024-02-10T05:57:32.856229+0000","last_unstale":"2024-02-10T05:58:15.016460+0000","last_undegraded":"2024-02-10T05:58:15.016460+0000","last_fullsized":"2024-02-10T05:58:15.016460+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:34.214983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"70'21","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367048+0000","last_change":"2024-02-10T05:57:35.045357+0000","last_active":"2024-02-10T05:58:15.367048+0000","last_peered":"2024-02-10T05:58:15.367048+0000","last_clean":"2024-02-10T05:58:15.367048+0000","last_became_active":"2024-02-10T05:57:35.044976+0000","last_became_peered":"2024-02-10T05:57:35.044976+0000","last_unstale":"2024-02-10T05:58:15.367048+0000","last_undegraded":"2024-02-10T05:58:15.367048+0000","last_fullsized":"2024-02-10T05:58:15.367048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:44.398663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016718+0000","last_change":"2024-02-10T05:57:38.947904+0000","last_active":"2024-02-10T05:58:15.016718+0000","last_peered":"2024-02-10T05:58:15.016718+0000","last_clean":"2024-02-10T05:58:15.016718+0000","last_became_active":"2024-02-10T05:57:38.947597+0000","last_became_peered":"2024-02-10T05:57:38.947597+0000","last_unstale":"2024-02-10T05:58:15.016718+0000","last_undegraded":"2024-02-10T05:58:15.016718+0000","last_fullsized":"2024-02-10T05:58:15.016718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:49.044062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006519+0000","last_change":"2024-02-10T05:57:36.904166+0000","last_active":"2024-02-10T05:58:15.006519+0000","last_peered":"2024-02-10T05:58:15.006519+0000","last_clean":"2024-02-10T05:58:15.006519+0000","last_became_active":"2024-02-10T05:57:36.903866+0000","last_became_peered":"2024-02-10T05:57:36.903866+0000","last_unstale":"2024-02-10T05:58:15.006519+0000","last_undegraded":"2024-02-10T05:58:15.006519+0000","last_fullsized":"2024-02-10T05:58:15.006519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:45.826388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.897762+0000","last_change":"2024-02-10T05:57:32.864171+0000","last_active":"2024-02-10T05:58:14.897762+0000","last_peered":"2024-02-10T05:58:14.897762+0000","last_clean":"2024-02-10T05:58:14.897762+0000","last_became_active":"2024-02-10T05:57:32.863648+0000","last_became_peered":"2024-02-10T05:57:32.863648+0000","last_unstale":"2024-02-10T05:58:14.897762+0000","last_undegraded":"2024-02-10T05:58:14.897762+0000","last_fullsized":"2024-02-10T05:58:14.897762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:36:33.391451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"70'12","reported_seq":66,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902460+0000","last_change":"2024-02-10T05:57:53.835070+0000","last_active":"2024-02-10T05:58:14.902460+0000","last_peered":"2024-02-10T05:58:14.902460+0000","last_clean":"2024-02-10T05:58:14.902460+0000","last_became_active":"2024-02-10T05:57:53.828730+0000","last_became_peered":"2024-02-10T05:57:53.828730+0000","last_unstale":"2024-02-10T05:58:14.902460+0000","last_undegraded":"2024-02-10T05:58:14.902460+0000","last_fullsized":"2024-02-10T05:58:14.902460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:05:02.109147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"70'1","reported_seq":22,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006147+0000","last_change":"2024-02-10T05:57:38.946775+0000","last_active":"2024-02-10T05:58:15.006147+0000","last_peered":"2024-02-10T05:58:15.006147+0000","last_clean":"2024-02-10T05:58:15.006147+0000","last_became_active":"2024-02-10T05:57:38.946496+0000","last_became_peered":"2024-02-10T05:57:38.946496+0000","last_unstale":"2024-02-10T05:58:15.006147+0000","last_undegraded":"2024-02-10T05:58:15.006147+0000","last_fullsized":"2024-02-10T05:58:15.006147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:37:04.535451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"63'2","reported_seq":45,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017139+0000","last_change":"2024-02-10T05:57:32.861323+0000","last_active":"2024-02-10T05:58:15.017139+0000","last_peered":"2024-02-10T05:58:15.017139+0000","last_clean":"2024-02-10T05:58:15.017139+0000","last_became_active":"2024-02-10T05:57:32.861045+0000","last_became_peered":"2024-02-10T05:57:32.861045+0000","last_unstale":"2024-02-10T05:58:15.017139+0000","last_undegraded":"2024-02-10T05:58:15.017139+0000","last_fullsized":"2024-02-10T05:58:15.017139+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:19:05.242990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"70'27","reported_seq":78,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005834+0000","last_change":"2024-02-10T05:57:35.046420+0000","last_active":"2024-02-10T05:58:15.005834+0000","last_peered":"2024-02-10T05:58:15.005834+0000","last_clean":"2024-02-10T05:58:15.005834+0000","last_became_active":"2024-02-10T05:57:35.045304+0000","last_became_peered":"2024-02-10T05:57:35.045304+0000","last_unstale":"2024-02-10T05:58:15.005834+0000","last_undegraded":"2024-02-10T05:58:15.005834+0000","last_fullsized":"2024-02-10T05:58:15.005834+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:25:12.720966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.200019+0000","last_change":"2024-02-10T05:57:36.905987+0000","last_active":"2024-02-10T05:58:43.200019+0000","last_peered":"2024-02-10T05:58:43.200019+0000","last_clean":"2024-02-10T05:58:43.200019+0000","last_became_active":"2024-02-10T05:57:36.905600+0000","last_became_peered":"2024-02-10T05:57:36.905600+0000","last_unstale":"2024-02-10T05:58:43.200019+0000","last_undegraded":"2024-02-10T05:58:43.200019+0000","last_fullsized":"2024-02-10T05:58:43.200019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:00:15.291461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903277+0000","last_change":"2024-02-10T05:57:38.914363+0000","last_active":"2024-02-10T05:58:14.903277+0000","last_peered":"2024-02-10T05:58:14.903277+0000","last_clean":"2024-02-10T05:58:14.903277+0000","last_became_active":"2024-02-10T05:57:38.914022+0000","last_became_peered":"2024-02-10T05:57:38.914022+0000","last_unstale":"2024-02-10T05:58:14.903277+0000","last_undegraded":"2024-02-10T05:58:14.903277+0000","last_fullsized":"2024-02-10T05:58:14.903277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:22:19.635165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'1","reported_seq":39,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017050+0000","last_change":"2024-02-10T05:57:32.856574+0000","last_active":"2024-02-10T05:58:15.017050+0000","last_peered":"2024-02-10T05:58:15.017050+0000","last_clean":"2024-02-10T05:58:15.017050+0000","last_became_active":"2024-02-10T05:57:32.856284+0000","last_became_peered":"2024-02-10T05:57:32.856284+0000","last_unstale":"2024-02-10T05:58:15.017050+0000","last_undegraded":"2024-02-10T05:58:15.017050+0000","last_fullsized":"2024-02-10T05:58:15.017050+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:52:33.757963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"70'24","reported_seq":76,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048695+0000","last_change":"2024-02-10T05:57:34.882038+0000","last_active":"2024-02-10T05:58:15.048695+0000","last_peered":"2024-02-10T05:58:15.048695+0000","last_clean":"2024-02-10T05:58:15.048695+0000","last_became_active":"2024-02-10T05:57:34.881103+0000","last_became_peered":"2024-02-10T05:57:34.881103+0000","last_unstale":"2024-02-10T05:58:15.048695+0000","last_undegraded":"2024-02-10T05:58:15.048695+0000","last_fullsized":"2024-02-10T05:58:15.048695+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:26:56.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013135+0000","last_change":"2024-02-10T05:57:36.902577+0000","last_active":"2024-02-10T05:58:15.013135+0000","last_peered":"2024-02-10T05:58:15.013135+0000","last_clean":"2024-02-10T05:58:15.013135+0000","last_became_active":"2024-02-10T05:57:36.902266+0000","last_became_peered":"2024-02-10T05:57:36.902266+0000","last_unstale":"2024-02-10T05:58:15.013135+0000","last_undegraded":"2024-02-10T05:58:15.013135+0000","last_fullsized":"2024-02-10T05:58:15.013135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:53:06.292543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016242+0000","last_change":"2024-02-10T05:57:38.910628+0000","last_active":"2024-02-10T05:58:15.016242+0000","last_peered":"2024-02-10T05:58:15.016242+0000","last_clean":"2024-02-10T05:58:15.016242+0000","last_became_active":"2024-02-10T05:57:38.910355+0000","last_became_peered":"2024-02-10T05:57:38.910355+0000","last_unstale":"2024-02-10T05:58:15.016242+0000","last_undegraded":"2024-02-10T05:58:15.016242+0000","last_fullsized":"2024-02-10T05:58:15.016242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:58:14.050942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013325+0000","last_change":"2024-02-10T05:57:36.903706+0000","last_active":"2024-02-10T05:58:15.013325+0000","last_peered":"2024-02-10T05:58:15.013325+0000","last_clean":"2024-02-10T05:58:15.013325+0000","last_became_active":"2024-02-10T05:57:36.903387+0000","last_became_peered":"2024-02-10T05:57:36.903387+0000","last_unstale":"2024-02-10T05:58:15.013325+0000","last_undegraded":"2024-02-10T05:58:15.013325+0000","last_fullsized":"2024-02-10T05:58:15.013325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:36:42.003559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016986+0000","last_change":"2024-02-10T05:57:32.860816+0000","last_active":"2024-02-10T05:58:15.016986+0000","last_peered":"2024-02-10T05:58:15.016986+0000","last_clean":"2024-02-10T05:58:15.016986+0000","last_became_active":"2024-02-10T05:57:32.860493+0000","last_became_peered":"2024-02-10T05:57:32.860493+0000","last_unstale":"2024-02-10T05:58:15.016986+0000","last_undegraded":"2024-02-10T05:58:15.016986+0000","last_fullsized":"2024-02-10T05:58:15.016986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:57:12.035724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"70'25","reported_seq":72,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898107+0000","last_change":"2024-02-10T05:57:35.045615+0000","last_active":"2024-02-10T05:58:14.898107+0000","last_peered":"2024-02-10T05:58:14.898107+0000","last_clean":"2024-02-10T05:58:14.898107+0000","last_became_active":"2024-02-10T05:57:35.045247+0000","last_became_peered":"2024-02-10T05:57:35.045247+0000","last_unstale":"2024-02-10T05:58:14.898107+0000","last_undegraded":"2024-02-10T05:58:14.898107+0000","last_fullsized":"2024-02-10T05:58:14.898107+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:40:31.028585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367214+0000","last_change":"2024-02-10T05:57:38.907822+0000","last_active":"2024-02-10T05:58:15.367214+0000","last_peered":"2024-02-10T05:58:15.367214+0000","last_clean":"2024-02-10T05:58:15.367214+0000","last_became_active":"2024-02-10T05:57:38.907510+0000","last_became_peered":"2024-02-10T05:57:38.907510+0000","last_unstale":"2024-02-10T05:58:15.367214+0000","last_undegraded":"2024-02-10T05:58:15.367214+0000","last_fullsized":"2024-02-10T05:58:15.367214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:36:55.851283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013344+0000","last_change":"2024-02-10T05:57:36.894481+0000","last_active":"2024-02-10T05:58:15.013344+0000","last_peered":"2024-02-10T05:58:15.013344+0000","last_clean":"2024-02-10T05:58:15.013344+0000","last_became_active":"2024-02-10T05:57:36.894105+0000","last_became_peered":"2024-02-10T05:57:36.894105+0000","last_unstale":"2024-02-10T05:58:15.013344+0000","last_undegraded":"2024-02-10T05:58:15.013344+0000","last_fullsized":"2024-02-10T05:58:15.013344+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:16:35.789344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006244+0000","last_change":"2024-02-10T05:57:32.865967+0000","last_active":"2024-02-10T05:58:15.006244+0000","last_peered":"2024-02-10T05:58:15.006244+0000","last_clean":"2024-02-10T05:58:15.006244+0000","last_became_active":"2024-02-10T05:57:32.865656+0000","last_became_peered":"2024-02-10T05:57:32.865656+0000","last_unstale":"2024-02-10T05:58:15.006244+0000","last_undegraded":"2024-02-10T05:58:15.006244+0000","last_fullsized":"2024-02-10T05:58:15.006244+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:45:07.315244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"70'15","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367970+0000","last_change":"2024-02-10T05:57:34.881314+0000","last_active":"2024-02-10T05:58:15.367970+0000","last_peered":"2024-02-10T05:58:15.367970+0000","last_clean":"2024-02-10T05:58:15.367970+0000","last_became_active":"2024-02-10T05:57:34.881024+0000","last_became_peered":"2024-02-10T05:57:34.881024+0000","last_unstale":"2024-02-10T05:58:15.367970+0000","last_undegraded":"2024-02-10T05:58:15.367970+0000","last_fullsized":"2024-02-10T05:58:15.367970+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:24:55.650702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902669+0000","last_change":"2024-02-10T05:57:38.912952+0000","last_active":"2024-02-10T05:58:14.902669+0000","last_peered":"2024-02-10T05:58:14.902669+0000","last_clean":"2024-02-10T05:58:14.902669+0000","last_became_active":"2024-02-10T05:57:38.911996+0000","last_became_peered":"2024-02-10T05:57:38.911996+0000","last_unstale":"2024-02-10T05:58:14.902669+0000","last_undegraded":"2024-02-10T05:58:14.902669+0000","last_fullsized":"2024-02-10T05:58:14.902669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:23:25.541373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.005971+0000","last_change":"2024-02-10T05:57:35.893626+0000","last_active":"2024-02-10T05:58:15.005971+0000","last_peered":"2024-02-10T05:58:15.005971+0000","last_clean":"2024-02-10T05:58:15.005971+0000","last_became_active":"2024-02-10T05:57:31.851364+0000","last_became_peered":"2024-02-10T05:57:31.851364+0000","last_unstale":"2024-02-10T05:58:15.005971+0000","last_undegraded":"2024-02-10T05:58:15.005971+0000","last_fullsized":"2024-02-10T05:58:15.005971+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:05:27.002023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020614230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048612+0000","last_change":"2024-02-10T05:57:32.865272+0000","last_active":"2024-02-10T05:58:15.048612+0000","last_peered":"2024-02-10T05:58:15.048612+0000","last_clean":"2024-02-10T05:58:15.048612+0000","last_became_active":"2024-02-10T05:57:32.864997+0000","last_became_peered":"2024-02-10T05:57:32.864997+0000","last_unstale":"2024-02-10T05:58:15.048612+0000","last_undegraded":"2024-02-10T05:58:15.048612+0000","last_fullsized":"2024-02-10T05:58:15.048612+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:29:36.850704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"70'34","reported_seq":105,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199382+0000","last_change":"2024-02-10T05:57:34.881001+0000","last_active":"2024-02-10T05:58:43.199382+0000","last_peered":"2024-02-10T05:58:43.199382+0000","last_clean":"2024-02-10T05:58:43.199382+0000","last_became_active":"2024-02-10T05:57:34.880701+0000","last_became_peered":"2024-02-10T05:57:34.880701+0000","last_unstale":"2024-02-10T05:58:43.199382+0000","last_undegraded":"2024-02-10T05:58:43.199382+0000","last_fullsized":"2024-02-10T05:58:43.199382+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:00:32.434882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367602+0000","last_change":"2024-02-10T05:57:36.907335+0000","last_active":"2024-02-10T05:58:15.367602+0000","last_peered":"2024-02-10T05:58:15.367602+0000","last_clean":"2024-02-10T05:58:15.367602+0000","last_became_active":"2024-02-10T05:57:36.905411+0000","last_became_peered":"2024-02-10T05:57:36.905411+0000","last_unstale":"2024-02-10T05:58:15.367602+0000","last_undegraded":"2024-02-10T05:58:15.367602+0000","last_fullsized":"2024-02-10T05:58:15.367602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:21:32.554532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.017088+0000","last_change":"2024-02-10T05:57:38.946373+0000","last_active":"2024-02-10T05:58:15.017088+0000","last_peered":"2024-02-10T05:58:15.017088+0000","last_clean":"2024-02-10T05:58:15.017088+0000","last_became_active":"2024-02-10T05:57:38.946084+0000","last_became_peered":"2024-02-10T05:57:38.946084+0000","last_unstale":"2024-02-10T05:58:15.017088+0000","last_undegraded":"2024-02-10T05:58:15.017088+0000","last_fullsized":"2024-02-10T05:58:15.017088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:25:52.027395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901215+0000","last_change":"2024-02-10T05:57:33.146153+0000","last_active":"2024-02-10T05:58:14.901215+0000","last_peered":"2024-02-10T05:58:14.901215+0000","last_clean":"2024-02-10T05:58:14.901215+0000","last_became_active":"2024-02-10T05:57:33.145791+0000","last_became_peered":"2024-02-10T05:57:33.145791+0000","last_unstale":"2024-02-10T05:58:14.901215+0000","last_undegraded":"2024-02-10T05:58:14.901215+0000","last_fullsized":"2024-02-10T05:58:14.901215+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:06:17.827042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013182+0000","last_change":"2024-02-10T05:57:35.886227+0000","last_active":"2024-02-10T05:58:15.013182+0000","last_peered":"2024-02-10T05:58:15.013182+0000","last_clean":"2024-02-10T05:58:15.013182+0000","last_became_active":"2024-02-10T05:57:31.855612+0000","last_became_peered":"2024-02-10T05:57:31.855612+0000","last_unstale":"2024-02-10T05:58:15.013182+0000","last_undegraded":"2024-02-10T05:58:15.013182+0000","last_fullsized":"2024-02-10T05:58:15.013182+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:16:14.145699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044142500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"70'20","reported_seq":69,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902351+0000","last_change":"2024-02-10T05:57:34.878287+0000","last_active":"2024-02-10T05:58:14.902351+0000","last_peered":"2024-02-10T05:58:14.902351+0000","last_clean":"2024-02-10T05:58:14.902351+0000","last_became_active":"2024-02-10T05:57:34.877948+0000","last_became_peered":"2024-02-10T05:57:34.877948+0000","last_unstale":"2024-02-10T05:58:14.902351+0000","last_undegraded":"2024-02-10T05:58:14.902351+0000","last_fullsized":"2024-02-10T05:58:14.902351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:52.387764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013441+0000","last_change":"2024-02-10T05:57:36.901677+0000","last_active":"2024-02-10T05:58:15.013441+0000","last_peered":"2024-02-10T05:58:15.013441+0000","last_clean":"2024-02-10T05:58:15.013441+0000","last_became_active":"2024-02-10T05:57:36.901404+0000","last_became_peered":"2024-02-10T05:57:36.901404+0000","last_unstale":"2024-02-10T05:58:15.013441+0000","last_undegraded":"2024-02-10T05:58:15.013441+0000","last_fullsized":"2024-02-10T05:58:15.013441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:21:04.800761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006522+0000","last_change":"2024-02-10T05:57:38.910382+0000","last_active":"2024-02-10T05:58:15.006522+0000","last_peered":"2024-02-10T05:58:15.006522+0000","last_clean":"2024-02-10T05:58:15.006522+0000","last_became_active":"2024-02-10T05:57:38.910113+0000","last_became_peered":"2024-02-10T05:57:38.910113+0000","last_unstale":"2024-02-10T05:58:15.006522+0000","last_undegraded":"2024-02-10T05:58:15.006522+0000","last_fullsized":"2024-02-10T05:58:15.006522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:59:59.107005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'128","reported_seq":279,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:17.010754+0000","last_change":"2024-02-10T05:57:23.349116+0000","last_active":"2024-02-10T05:58:17.010754+0000","last_peered":"2024-02-10T05:58:17.010754+0000","last_clean":"2024-02-10T05:58:17.010754+0000","last_became_active":"2024-02-10T05:57:23.348655+0000","last_became_peered":"2024-02-10T05:57:23.348655+0000","last_unstale":"2024-02-10T05:58:17.010754+0000","last_undegraded":"2024-02-10T05:58:17.010754+0000","last_fullsized":"2024-02-10T05:58:17.010754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:13.055885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:13.055885+0000","last_clean_scrub_stamp":"2024-02-10T05:55:13.055885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:35:37.841146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902710+0000","last_change":"2024-02-10T05:57:32.857101+0000","last_active":"2024-02-10T05:58:14.902710+0000","last_peered":"2024-02-10T05:58:14.902710+0000","last_clean":"2024-02-10T05:58:14.902710+0000","last_became_active":"2024-02-10T05:57:32.856650+0000","last_became_peered":"2024-02-10T05:57:32.856650+0000","last_unstale":"2024-02-10T05:58:14.902710+0000","last_undegraded":"2024-02-10T05:58:14.902710+0000","last_fullsized":"2024-02-10T05:58:14.902710+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:31:42.478191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"70'16","reported_seq":67,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199800+0000","last_change":"2024-02-10T05:57:35.045646+0000","last_active":"2024-02-10T05:58:43.199800+0000","last_peered":"2024-02-10T05:58:43.199800+0000","last_clean":"2024-02-10T05:58:43.199800+0000","last_became_active":"2024-02-10T05:57:35.045307+0000","last_became_peered":"2024-02-10T05:57:35.045307+0000","last_unstale":"2024-02-10T05:58:43.199800+0000","last_undegraded":"2024-02-10T05:58:43.199800+0000","last_fullsized":"2024-02-10T05:58:43.199800+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:00:26.052403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016660+0000","last_change":"2024-02-10T05:57:36.905906+0000","last_active":"2024-02-10T05:58:15.016660+0000","last_peered":"2024-02-10T05:58:15.016660+0000","last_clean":"2024-02-10T05:58:15.016660+0000","last_became_active":"2024-02-10T05:57:36.905620+0000","last_became_peered":"2024-02-10T05:57:36.905620+0000","last_unstale":"2024-02-10T05:58:15.016660+0000","last_undegraded":"2024-02-10T05:58:15.016660+0000","last_fullsized":"2024-02-10T05:58:15.016660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:54:37.104479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901743+0000","last_change":"2024-02-10T05:57:38.910473+0000","last_active":"2024-02-10T05:58:14.901743+0000","last_peered":"2024-02-10T05:58:14.901743+0000","last_clean":"2024-02-10T05:58:14.901743+0000","last_became_active":"2024-02-10T05:57:38.910162+0000","last_became_peered":"2024-02-10T05:57:38.910162+0000","last_unstale":"2024-02-10T05:58:14.901743+0000","last_undegraded":"2024-02-10T05:58:14.901743+0000","last_fullsized":"2024-02-10T05:58:14.901743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:27:22.350693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367438+0000","last_change":"2024-02-10T05:57:32.859956+0000","last_active":"2024-02-10T05:58:15.367438+0000","last_peered":"2024-02-10T05:58:15.367438+0000","last_clean":"2024-02-10T05:58:15.367438+0000","last_became_active":"2024-02-10T05:57:32.859543+0000","last_became_peered":"2024-02-10T05:57:32.859543+0000","last_unstale":"2024-02-10T05:58:15.367438+0000","last_undegraded":"2024-02-10T05:58:15.367438+0000","last_fullsized":"2024-02-10T05:58:15.367438+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:33.880345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"71'5","reported_seq":116,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:49.863489+0000","last_change":"2024-02-10T05:57:35.885973+0000","last_active":"2024-02-10T05:58:49.863489+0000","last_peered":"2024-02-10T05:58:49.863489+0000","last_clean":"2024-02-10T05:58:49.863489+0000","last_became_active":"2024-02-10T05:57:31.853279+0000","last_became_peered":"2024-02-10T05:57:31.853279+0000","last_unstale":"2024-02-10T05:58:49.863489+0000","last_undegraded":"2024-02-10T05:58:49.863489+0000","last_fullsized":"2024-02-10T05:58:49.863489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:30.836127+0000","last_clean_scrub_stamp":"2024-02-10T05:57:30.836127+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:02:06.901598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000371016,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"70'12","reported_seq":47,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367494+0000","last_change":"2024-02-10T05:57:34.882177+0000","last_active":"2024-02-10T05:58:15.367494+0000","last_peered":"2024-02-10T05:58:15.367494+0000","last_clean":"2024-02-10T05:58:15.367494+0000","last_became_active":"2024-02-10T05:57:34.881898+0000","last_became_peered":"2024-02-10T05:57:34.881898+0000","last_unstale":"2024-02-10T05:58:15.367494+0000","last_undegraded":"2024-02-10T05:58:15.367494+0000","last_fullsized":"2024-02-10T05:58:15.367494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:19:12.070557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006707+0000","last_change":"2024-02-10T05:57:36.906274+0000","last_active":"2024-02-10T05:58:15.006707+0000","last_peered":"2024-02-10T05:58:15.006707+0000","last_clean":"2024-02-10T05:58:15.006707+0000","last_became_active":"2024-02-10T05:57:36.905576+0000","last_became_peered":"2024-02-10T05:57:36.905576+0000","last_unstale":"2024-02-10T05:58:15.006707+0000","last_undegraded":"2024-02-10T05:58:15.006707+0000","last_fullsized":"2024-02-10T05:58:15.006707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:25:53.282213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901098+0000","last_change":"2024-02-10T05:57:38.946709+0000","last_active":"2024-02-10T05:58:14.901098+0000","last_peered":"2024-02-10T05:58:14.901098+0000","last_clean":"2024-02-10T05:58:14.901098+0000","last_became_active":"2024-02-10T05:57:38.946393+0000","last_became_peered":"2024-02-10T05:57:38.946393+0000","last_unstale":"2024-02-10T05:58:14.901098+0000","last_undegraded":"2024-02-10T05:58:14.901098+0000","last_fullsized":"2024-02-10T05:58:14.901098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:15:07.035953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901866+0000","last_change":"2024-02-10T05:57:33.146640+0000","last_active":"2024-02-10T05:58:14.901866+0000","last_peered":"2024-02-10T05:58:14.901866+0000","last_clean":"2024-02-10T05:58:14.901866+0000","last_became_active":"2024-02-10T05:57:33.146099+0000","last_became_peered":"2024-02-10T05:57:33.146099+0000","last_unstale":"2024-02-10T05:58:14.901866+0000","last_undegraded":"2024-02-10T05:58:14.901866+0000","last_fullsized":"2024-02-10T05:58:14.901866+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:49.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"70'25","reported_seq":77,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367012+0000","last_change":"2024-02-10T05:57:35.045234+0000","last_active":"2024-02-10T05:58:15.367012+0000","last_peered":"2024-02-10T05:58:15.367012+0000","last_clean":"2024-02-10T05:58:15.367012+0000","last_became_active":"2024-02-10T05:57:35.044942+0000","last_became_peered":"2024-02-10T05:57:35.044942+0000","last_unstale":"2024-02-10T05:58:15.367012+0000","last_undegraded":"2024-02-10T05:58:15.367012+0000","last_fullsized":"2024-02-10T05:58:15.367012+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:55:45.304417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898231+0000","last_change":"2024-02-10T05:57:36.898973+0000","last_active":"2024-02-10T05:58:14.898231+0000","last_peered":"2024-02-10T05:58:14.898231+0000","last_clean":"2024-02-10T05:58:14.898231+0000","last_became_active":"2024-02-10T05:57:36.898676+0000","last_became_peered":"2024-02-10T05:57:36.898676+0000","last_unstale":"2024-02-10T05:58:14.898231+0000","last_undegraded":"2024-02-10T05:58:14.898231+0000","last_fullsized":"2024-02-10T05:58:14.898231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:40:40.205926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367126+0000","last_change":"2024-02-10T05:57:38.911648+0000","last_active":"2024-02-10T05:58:15.367126+0000","last_peered":"2024-02-10T05:58:15.367126+0000","last_clean":"2024-02-10T05:58:15.367126+0000","last_became_active":"2024-02-10T05:57:38.911369+0000","last_became_peered":"2024-02-10T05:57:38.911369+0000","last_unstale":"2024-02-10T05:58:15.367126+0000","last_undegraded":"2024-02-10T05:58:15.367126+0000","last_fullsized":"2024-02-10T05:58:15.367126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:03:59.716478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"63'1","reported_seq":39,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006317+0000","last_change":"2024-02-10T05:57:33.145375+0000","last_active":"2024-02-10T05:58:15.006317+0000","last_peered":"2024-02-10T05:58:15.006317+0000","last_clean":"2024-02-10T05:58:15.006317+0000","last_became_active":"2024-02-10T05:57:33.145028+0000","last_became_peered":"2024-02-10T05:57:33.145028+0000","last_unstale":"2024-02-10T05:58:15.006317+0000","last_undegraded":"2024-02-10T05:58:15.006317+0000","last_fullsized":"2024-02-10T05:58:15.006317+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:14:57.252610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"70'16","reported_seq":56,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.901572+0000","last_change":"2024-02-10T05:57:34.879043+0000","last_active":"2024-02-10T05:58:14.901572+0000","last_peered":"2024-02-10T05:58:14.901572+0000","last_clean":"2024-02-10T05:58:14.901572+0000","last_became_active":"2024-02-10T05:57:34.878435+0000","last_became_peered":"2024-02-10T05:57:34.878435+0000","last_unstale":"2024-02-10T05:58:14.901572+0000","last_undegraded":"2024-02-10T05:58:14.901572+0000","last_fullsized":"2024-02-10T05:58:14.901572+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:49:49.966790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199378+0000","last_change":"2024-02-10T05:57:36.903103+0000","last_active":"2024-02-10T05:58:43.199378+0000","last_peered":"2024-02-10T05:58:43.199378+0000","last_clean":"2024-02-10T05:58:43.199378+0000","last_became_active":"2024-02-10T05:57:36.902748+0000","last_became_peered":"2024-02-10T05:57:36.902748+0000","last_unstale":"2024-02-10T05:58:43.199378+0000","last_undegraded":"2024-02-10T05:58:43.199378+0000","last_fullsized":"2024-02-10T05:58:43.199378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:11:09.134559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.016661+0000","last_change":"2024-02-10T05:57:38.948965+0000","last_active":"2024-02-10T05:58:15.016661+0000","last_peered":"2024-02-10T05:58:15.016661+0000","last_clean":"2024-02-10T05:58:15.016661+0000","last_became_active":"2024-02-10T05:57:38.948697+0000","last_became_peered":"2024-02-10T05:57:38.948697+0000","last_unstale":"2024-02-10T05:58:15.016661+0000","last_undegraded":"2024-02-10T05:58:15.016661+0000","last_fullsized":"2024-02-10T05:58:15.016661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:30.600684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":34,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.367625+0000","last_change":"2024-02-10T05:57:32.858246+0000","last_active":"2024-02-10T05:58:15.367625+0000","last_peered":"2024-02-10T05:58:15.367625+0000","last_clean":"2024-02-10T05:58:15.367625+0000","last_became_active":"2024-02-10T05:57:32.857823+0000","last_became_peered":"2024-02-10T05:57:32.857823+0000","last_unstale":"2024-02-10T05:58:15.367625+0000","last_undegraded":"2024-02-10T05:58:15.367625+0000","last_fullsized":"2024-02-10T05:58:15.367625+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:02:28.187420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"70'22","reported_seq":64,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048622+0000","last_change":"2024-02-10T05:57:34.872173+0000","last_active":"2024-02-10T05:58:15.048622+0000","last_peered":"2024-02-10T05:58:15.048622+0000","last_clean":"2024-02-10T05:58:15.048622+0000","last_became_active":"2024-02-10T05:57:34.871826+0000","last_became_peered":"2024-02-10T05:57:34.871826+0000","last_unstale":"2024-02-10T05:58:15.048622+0000","last_undegraded":"2024-02-10T05:58:15.048622+0000","last_fullsized":"2024-02-10T05:58:15.048622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:31:14.669348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199382+0000","last_change":"2024-02-10T05:57:36.901651+0000","last_active":"2024-02-10T05:58:43.199382+0000","last_peered":"2024-02-10T05:58:43.199382+0000","last_clean":"2024-02-10T05:58:43.199382+0000","last_became_active":"2024-02-10T05:57:36.901368+0000","last_became_peered":"2024-02-10T05:57:36.901368+0000","last_unstale":"2024-02-10T05:58:43.199382+0000","last_undegraded":"2024-02-10T05:58:43.199382+0000","last_fullsized":"2024-02-10T05:58:43.199382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:02.531853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048522+0000","last_change":"2024-02-10T05:57:38.909800+0000","last_active":"2024-02-10T05:58:15.048522+0000","last_peered":"2024-02-10T05:58:15.048522+0000","last_clean":"2024-02-10T05:58:15.048522+0000","last_became_active":"2024-02-10T05:57:38.909491+0000","last_became_peered":"2024-02-10T05:57:38.909491+0000","last_unstale":"2024-02-10T05:58:15.048522+0000","last_undegraded":"2024-02-10T05:58:15.048522+0000","last_fullsized":"2024-02-10T05:58:15.048522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:04:57.916809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902309+0000","last_change":"2024-02-10T05:57:32.861540+0000","last_active":"2024-02-10T05:58:14.902309+0000","last_peered":"2024-02-10T05:58:14.902309+0000","last_clean":"2024-02-10T05:58:14.902309+0000","last_became_active":"2024-02-10T05:57:32.861130+0000","last_became_peered":"2024-02-10T05:57:32.861130+0000","last_unstale":"2024-02-10T05:58:14.902309+0000","last_undegraded":"2024-02-10T05:58:14.902309+0000","last_fullsized":"2024-02-10T05:58:14.902309+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:45:27.247898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"70'30","reported_seq":85,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902791+0000","last_change":"2024-02-10T05:57:35.043917+0000","last_active":"2024-02-10T05:58:14.902791+0000","last_peered":"2024-02-10T05:58:14.902791+0000","last_clean":"2024-02-10T05:58:14.902791+0000","last_became_active":"2024-02-10T05:57:35.043616+0000","last_became_peered":"2024-02-10T05:57:35.043616+0000","last_unstale":"2024-02-10T05:58:14.902791+0000","last_undegraded":"2024-02-10T05:58:14.902791+0000","last_fullsized":"2024-02-10T05:58:14.902791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:15:31.903676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048435+0000","last_change":"2024-02-10T05:57:36.901508+0000","last_active":"2024-02-10T05:58:15.048435+0000","last_peered":"2024-02-10T05:58:15.048435+0000","last_clean":"2024-02-10T05:58:15.048435+0000","last_became_active":"2024-02-10T05:57:36.901249+0000","last_became_peered":"2024-02-10T05:57:36.901249+0000","last_unstale":"2024-02-10T05:58:15.048435+0000","last_undegraded":"2024-02-10T05:58:15.048435+0000","last_fullsized":"2024-02-10T05:58:15.048435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:51:38.104171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006121+0000","last_change":"2024-02-10T05:57:38.903424+0000","last_active":"2024-02-10T05:58:15.006121+0000","last_peered":"2024-02-10T05:58:15.006121+0000","last_clean":"2024-02-10T05:58:15.006121+0000","last_became_active":"2024-02-10T05:57:38.903197+0000","last_became_peered":"2024-02-10T05:57:38.903197+0000","last_unstale":"2024-02-10T05:58:15.006121+0000","last_undegraded":"2024-02-10T05:58:15.006121+0000","last_fullsized":"2024-02-10T05:58:15.006121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:27.767687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199735+0000","last_change":"2024-02-10T05:57:36.900643+0000","last_active":"2024-02-10T05:58:43.199735+0000","last_peered":"2024-02-10T05:58:43.199735+0000","last_clean":"2024-02-10T05:58:43.199735+0000","last_became_active":"2024-02-10T05:57:36.900358+0000","last_became_peered":"2024-02-10T05:57:36.900358+0000","last_unstale":"2024-02-10T05:58:43.199735+0000","last_undegraded":"2024-02-10T05:58:43.199735+0000","last_fullsized":"2024-02-10T05:58:43.199735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:08:53.572780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902707+0000","last_change":"2024-02-10T05:57:32.854246+0000","last_active":"2024-02-10T05:58:14.902707+0000","last_peered":"2024-02-10T05:58:14.902707+0000","last_clean":"2024-02-10T05:58:14.902707+0000","last_became_active":"2024-02-10T05:57:32.853756+0000","last_became_peered":"2024-02-10T05:57:32.853756+0000","last_unstale":"2024-02-10T05:58:14.902707+0000","last_undegraded":"2024-02-10T05:58:14.902707+0000","last_fullsized":"2024-02-10T05:58:14.902707+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:26:34.424373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"70'15","reported_seq":29,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013669+0000","last_change":"2024-02-10T05:57:53.837815+0000","last_active":"2024-02-10T05:58:15.013669+0000","last_peered":"2024-02-10T05:58:15.013669+0000","last_clean":"2024-02-10T05:58:15.013669+0000","last_became_active":"2024-02-10T05:57:53.826737+0000","last_became_peered":"2024-02-10T05:57:53.826737+0000","last_unstale":"2024-02-10T05:58:15.013669+0000","last_undegraded":"2024-02-10T05:58:15.013669+0000","last_fullsized":"2024-02-10T05:58:15.013669+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:38:29.273501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903039+0000","last_change":"2024-02-10T05:57:38.947474+0000","last_active":"2024-02-10T05:58:14.903039+0000","last_peered":"2024-02-10T05:58:14.903039+0000","last_clean":"2024-02-10T05:58:14.903039+0000","last_became_active":"2024-02-10T05:57:38.947048+0000","last_became_peered":"2024-02-10T05:57:38.947048+0000","last_unstale":"2024-02-10T05:58:14.903039+0000","last_undegraded":"2024-02-10T05:58:14.903039+0000","last_fullsized":"2024-02-10T05:58:14.903039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:34:06.693830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.006656+0000","last_change":"2024-02-10T05:57:36.899334+0000","last_active":"2024-02-10T05:58:15.006656+0000","last_peered":"2024-02-10T05:58:15.006656+0000","last_clean":"2024-02-10T05:58:15.006656+0000","last_became_active":"2024-02-10T05:57:36.898922+0000","last_became_peered":"2024-02-10T05:57:36.898922+0000","last_unstale":"2024-02-10T05:58:15.006656+0000","last_undegraded":"2024-02-10T05:58:15.006656+0000","last_fullsized":"2024-02-10T05:58:15.006656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:02.111313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048501+0000","last_change":"2024-02-10T05:57:33.145437+0000","last_active":"2024-02-10T05:58:15.048501+0000","last_peered":"2024-02-10T05:58:15.048501+0000","last_clean":"2024-02-10T05:58:15.048501+0000","last_became_active":"2024-02-10T05:57:33.145143+0000","last_became_peered":"2024-02-10T05:57:33.145143+0000","last_unstale":"2024-02-10T05:58:15.048501+0000","last_undegraded":"2024-02-10T05:58:15.048501+0000","last_fullsized":"2024-02-10T05:58:15.048501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:25:55.824341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"70'17","reported_seq":60,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049060+0000","last_change":"2024-02-10T05:57:34.881918+0000","last_active":"2024-02-10T05:58:15.049060+0000","last_peered":"2024-02-10T05:58:15.049060+0000","last_clean":"2024-02-10T05:58:15.049060+0000","last_became_active":"2024-02-10T05:57:34.880894+0000","last_became_peered":"2024-02-10T05:57:34.880894+0000","last_unstale":"2024-02-10T05:58:15.049060+0000","last_undegraded":"2024-02-10T05:58:15.049060+0000","last_fullsized":"2024-02-10T05:58:15.049060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:31:25.928060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.013810+0000","last_change":"2024-02-10T05:57:38.910919+0000","last_active":"2024-02-10T05:58:15.013810+0000","last_peered":"2024-02-10T05:58:15.013810+0000","last_clean":"2024-02-10T05:58:15.013810+0000","last_became_active":"2024-02-10T05:57:38.910602+0000","last_became_peered":"2024-02-10T05:57:38.910602+0000","last_unstale":"2024-02-10T05:58:15.013810+0000","last_undegraded":"2024-02-10T05:58:15.013810+0000","last_fullsized":"2024-02-10T05:58:15.013810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:27:27.004207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.902175+0000","last_change":"2024-02-10T05:57:36.901504+0000","last_active":"2024-02-10T05:58:14.902175+0000","last_peered":"2024-02-10T05:58:14.902175+0000","last_clean":"2024-02-10T05:58:14.902175+0000","last_became_active":"2024-02-10T05:57:36.901215+0000","last_became_peered":"2024-02-10T05:57:36.901215+0000","last_unstale":"2024-02-10T05:58:14.902175+0000","last_undegraded":"2024-02-10T05:58:14.902175+0000","last_fullsized":"2024-02-10T05:58:14.902175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:00:28.028821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.898131+0000","last_change":"2024-02-10T05:57:32.860729+0000","last_active":"2024-02-10T05:58:14.898131+0000","last_peered":"2024-02-10T05:58:14.898131+0000","last_clean":"2024-02-10T05:58:14.898131+0000","last_became_active":"2024-02-10T05:57:32.860426+0000","last_became_peered":"2024-02-10T05:57:32.860426+0000","last_unstale":"2024-02-10T05:58:14.898131+0000","last_undegraded":"2024-02-10T05:58:14.898131+0000","last_fullsized":"2024-02-10T05:58:14.898131+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:48:05.831815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"70'17","reported_seq":57,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049265+0000","last_change":"2024-02-10T05:57:34.881319+0000","last_active":"2024-02-10T05:58:15.049265+0000","last_peered":"2024-02-10T05:58:15.049265+0000","last_clean":"2024-02-10T05:58:15.049265+0000","last_became_active":"2024-02-10T05:57:34.880995+0000","last_became_peered":"2024-02-10T05:57:34.880995+0000","last_unstale":"2024-02-10T05:58:15.049265+0000","last_undegraded":"2024-02-10T05:58:15.049265+0000","last_fullsized":"2024-02-10T05:58:15.049265+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:45:40.948863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.049227+0000","last_change":"2024-02-10T05:57:38.904960+0000","last_active":"2024-02-10T05:58:15.049227+0000","last_peered":"2024-02-10T05:58:15.049227+0000","last_clean":"2024-02-10T05:58:15.049227+0000","last_became_active":"2024-02-10T05:57:38.904757+0000","last_became_peered":"2024-02-10T05:57:38.904757+0000","last_unstale":"2024-02-10T05:58:15.049227+0000","last_undegraded":"2024-02-10T05:58:15.049227+0000","last_fullsized":"2024-02-10T05:58:15.049227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:37.884463+0000","last_clean_scrub_stamp":"2024-02-10T05:57:37.884463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:36:30.214057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"70'8","reported_seq":49,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:43.199383+0000","last_change":"2024-02-10T05:57:36.900932+0000","last_active":"2024-02-10T05:58:43.199383+0000","last_peered":"2024-02-10T05:58:43.199383+0000","last_clean":"2024-02-10T05:58:43.199383+0000","last_became_active":"2024-02-10T05:57:36.900500+0000","last_became_peered":"2024-02-10T05:57:36.900500+0000","last_unstale":"2024-02-10T05:58:43.199383+0000","last_undegraded":"2024-02-10T05:58:43.199383+0000","last_fullsized":"2024-02-10T05:58:43.199383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:35.871695+0000","last_clean_scrub_stamp":"2024-02-10T05:57:35.871695+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:15.439510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:14.903398+0000","last_change":"2024-02-10T05:57:32.860898+0000","last_active":"2024-02-10T05:58:14.903398+0000","last_peered":"2024-02-10T05:58:14.903398+0000","last_clean":"2024-02-10T05:58:14.903398+0000","last_became_active":"2024-02-10T05:57:32.860558+0000","last_became_peered":"2024-02-10T05:57:32.860558+0000","last_unstale":"2024-02-10T05:58:14.903398+0000","last_undegraded":"2024-02-10T05:58:14.903398+0000","last_fullsized":"2024-02-10T05:58:14.903398+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:31.842984+0000","last_clean_scrub_stamp":"2024-02-10T05:57:31.842984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:20:10.382846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"70'10","reported_seq":44,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-10T05:58:15.048899+0000","last_change":"2024-02-10T05:57:34.874742+0000","last_active":"2024-02-10T05:58:15.048899+0000","last_peered":"2024-02-10T05:58:15.048899+0000","last_clean":"2024-02-10T05:58:15.048899+0000","last_became_active":"2024-02-10T05:57:34.874379+0000","last_became_peered":"2024-02-10T05:57:34.874379+0000","last_unstale":"2024-02-10T05:58:15.048899+0000","last_undegraded":"2024-02-10T05:58:15.048899+0000","last_fullsized":"2024-02-10T05:58:15.048899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:57:33.855647+0000","last_clean_scrub_stamp":"2024-02-10T05:57:33.855647+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:28.431131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":591,"ondisk_log_size":591,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168597,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32568,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712584,"statfs":{"total":95995035648,"available":95961686016,"internally_reserved":0,"allocated":1425408,"data_stored":937454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.496},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.57999999999999996}]},{"osd":4,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.60799999999999998}]},{"osd":5,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":5.891,"5min":5.891,"15min":5.891},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Feb 10 05:58:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.56200000000000006}]}]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1404928,"data_stored":936018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52000000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.436}]},{"osd":4,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.50700000000000001}]},{"osd":5,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":347600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.35099999999999998,"15min":0.32800000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.54500000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.36499999999999999,"15min":0.34000000000000002},"min":{"1min":0.28599999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.318}]},{"osd":2,"last update":"Sat Feb 10 05:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.53200000000000003,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46999999999999997,"15min":0.48099999999999998},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.438,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.34799999999999998}]},{"osd":4,"last update":"Sat Feb 10 05:58:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55500000000000005,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73199999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.5},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.751,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.64700000000000002}]},{"osd":5,"last update":"Sat Feb 10 05:58:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.48799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82099999999999995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.80000000000000004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Feb 10 05:58:19 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.62}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1429504,"data_stored":936812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.35799999999999998,"15min":0.33200000000000002},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.37,"15min":0.34999999999999998},"min":{"1min":0.215,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.66900000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:58:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.47199999999999998,"15min":0.432},"min":{"1min":0.26800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.7009999999999996,"5min":4.7009999999999996,"15min":4.7009999999999996},"last":0.375},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.47199999999999998,"15min":0.433},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.8339999999999996,"5min":4.8339999999999996,"15min":4.8339999999999996},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:58:43 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.50700000000000001,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.8019999999999996,"5min":4.8019999999999996,"15min":4.8019999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.5,"15min":0.46600000000000003},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.9699999999999998,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":0.42999999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:58:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.46500000000000002,"5min":0.443,"15min":0.443},"max":{"1min":0.70599999999999996,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.60199999999999998,"15min":0.60799999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":1.377,"15min":1.377},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Feb 10 05:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.64100000000000001,"15min":0.61299999999999999},"min":{"1min":0.45100000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.6699999999999999,"5min":4.6699999999999999,"15min":4.6699999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.65200000000000002,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9400000000000004,"5min":4.9400000000000004,"15min":4.9400000000000004},"last":0.53200000000000003}]},{"osd":6,"last update":"Sat Feb 10 05:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":4.8200000000000003,"5min":4.8200000000000003,"15min":4.8200000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":4.9880000000000004,"5min":4.9880000000000004,"15min":4.9880000000000004},"last":0.57899999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":346735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.39100000000000001,"15min":0.378},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.27000000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.40600000000000003}]},{"osd":1,"last update":"Sat Feb 10 05:58:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.39400000000000002,"15min":0.39000000000000001},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.54400000000000004,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.40000000000000002},"min":{"1min":0.26100000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Feb 10 05:58:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51300000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.443,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Feb 10 05:58:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Sat Feb 10 05:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57999999999999996,"15min":0.58999999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.67100000000000004,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.67600000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.81299999999999994}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.746}]}]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":847872,"data_stored":347402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.253,"5min":0.184,"15min":0.184},"max":{"1min":0.56000000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.42399999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.54800000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.311}]},{"osd":1,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.373,"15min":0.373},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.47099999999999997,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.23499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51500000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:58:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.42799999999999999,"15min":0.42899999999999999},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.57199999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.54500000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.39900000000000002}]},{"osd":4,"last update":"Sat Feb 10 05:58:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.64900000000000002,"15min":0.65600000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.4359999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69299999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.53700000000000003}]},{"osd":5,"last update":"Sat Feb 10 05:58:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78100000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.67700000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.626},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61499999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":839680,"data_stored":346640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.70099999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.67300000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72799999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.505,"15min":0.5},"min":{"1min":0.38400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.65200000000000002,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.621}]},{"osd":3,"last update":"Sat Feb 10 05:58:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Feb 10 05:58:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.40999999999999998,"15min":0.40300000000000002},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.41499999999999998,"15min":0.40300000000000002},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44}]},{"osd":6,"last update":"Sat Feb 10 05:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.497},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Feb 10 05:58:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.65000000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724575,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":823296,"data_stored":346959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.63300000000000001,"5min":0.872,"15min":0.872},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55400000000000005,"15min":0.55700000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71199999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68700000000000006}]},{"osd":1,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.70099999999999996,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.65200000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.39100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69599999999999995,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.72999999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Feb 10 05:58:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.308,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.42599999999999999,"15min":0.42199999999999999},"min":{"1min":0.29499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.63,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59699999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:57:56 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Feb 10 05:58:22 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.79000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:58:57.924 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T05:58:57.924 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T05:58:57.925 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T05:58:57.925 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T05:58:58.162 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-mon[41707]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:58.163 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-mon[41707]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:58.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-mon[30020]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:58.163 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-mon[30020]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:58.163 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[79124]: ts=2024-02-10T05:58:58.110Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T05:58:58.163 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:58:58 smithi121 bash[79098]: c1c2d4d0986ec15a53f6f09bb15bb849bd1da5d28a26740630c64f82bfce78d6 2024-02-10T05:58:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:58:58.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:58 smithi146 ceph-mon[33795]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:58:58.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:58 smithi146 ceph-mon[33795]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:58:58.432 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:58:58 smithi121 systemd[1]: Started Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE Bus STOPPING 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE Bus STOPPED 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE Bus STARTING 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE Serving on http://:::9283 2024-02-10T05:58:58.433 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:58:58 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:58:58] ENGINE Bus STARTED 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[41707]: Deploying daemon grafana.a on smithi146 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:58:59 smithi121 ceph-mon[30020]: Deploying daemon grafana.a on smithi146 2024-02-10T05:58:59.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:58:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:58:59 smithi146 ceph-mon[33795]: Deploying daemon grafana.a on smithi146 2024-02-10T05:59:00.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 -- 172.21.15.121:0/3991198534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc06ae30 msgr2=0x7f98dc10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:59:00.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 --2- 172.21.15.121:0/3991198534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc06ae30 0x7f98dc10eb70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b0a0 tx=0x7f98d400b3b0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 -- 172.21.15.121:0/3991198534 shutdown_connections 2024-02-10T05:59:00.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 --2- 172.21.15.121:0/3991198534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc06ae30 0x7f98dc10eb70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 --2- 172.21.15.121:0/3991198534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc107d50 0x7f98dc108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 --2- 172.21.15.121:0/3991198534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06a470 0x7f98dc06a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 -- 172.21.15.121:0/3991198534 >> 172.21.15.121:0/3991198534 conn(0x7f98dc077c30 msgr2=0x7f98dc078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 -- 172.21.15.121:0/3991198534 shutdown_connections 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.110+0000 7f98e2d86700 1 -- 172.21.15.121:0/3991198534 wait complete. 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 Processor -- start 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 -- start start 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc06a470 0x7f98dc100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 0x7f98dc101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1a5c30 con 0x7f98dc06ae30 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1a5da0 con 0x7f98dc06a470 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e2d86700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1a5f60 con 0x7f98dc107d50 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98dbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:59:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e1323700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 0x7f98dc101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98e0b22700 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc06a470 0x7f98dc100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98dbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59022/0 (socket says 172.21.15.121:59022) 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.111+0000 7f98dbfff700 1 -- 172.21.15.121:0/2491404041 learned_addr learned my addr 172.21.15.121:0/2491404041 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 -- 172.21.15.121:0/2491404041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc06a470 msgr2=0x7f98dc100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc06a470 0x7f98dc100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 -- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 msgr2=0x7f98dc101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 -- 172.21.15.121:0/2491404041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8009520 con 0x7f98dc107d50 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98dbfff700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e1323700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 0x7f98dc101750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b0a0 tx=0x7f98d40078e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d400e040 con 0x7f98dc107d50 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f98d401d470 con 0x7f98dc107d50 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d4009d00 con 0x7f98dc107d50 2024-02-10T05:59:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40052a0 con 0x7f98dc107d50 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.112+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc1a6510 con 0x7f98dc107d50 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.113+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc10a230 con 0x7f98dc107d50 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.113+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f98d4004a50 con 0x7f98dc107d50 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.114+0000 7f98d9ffb700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 0x7f98cc06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.114+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f98d408ea40 con 0x7f98dc107d50 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.114+0000 7f98e0b22700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 0x7f98cc06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.114+0000 7f98e0b22700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 0x7f98cc06ff80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009be0 tx=0x7f98d0009260 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T05:59:00.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.119+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98d4090e60 con 0x7f98dc107d50 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[41707]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[30020]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:00.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:00 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[79124]: ts=2024-02-10T05:59:00.076Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00013812s 2024-02-10T05:59:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.368+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98dc1112a0 con 0x7f98dc107d50 2024-02-10T05:59:00.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.368+0000 7f98d9ffb700 1 -- 172.21.15.121:0/2491404041 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f98d4057ed0 con 0x7f98dc107d50 2024-02-10T05:59:00.368 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-02-10T05:59:00.369 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T05:59:00.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 msgr2=0x7f98cc06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 0x7f98cc06ff80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009be0 tx=0x7f98d0009260 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 msgr2=0x7f98dc101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 0x7f98dc101750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f98d400b0a0 tx=0x7f98d40078e0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 shutdown_connections 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f98cc06dac0 0x7f98cc06ff80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98dc107d50 0x7f98dc101750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc06a470 0x7f98dc100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 --2- 172.21.15.121:0/2491404041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98dc06ae30 0x7f98dc101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 >> 172.21.15.121:0/2491404041 conn(0x7f98dc077c30 msgr2=0x7f98dc0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 shutdown_connections 2024-02-10T05:59:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-02-10T05:59:00.370+0000 7f98e2d86700 1 -- 172.21.15.121:0/2491404041 wait complete. 2024-02-10T05:59:00.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:00 smithi146 ceph-mon[33795]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:00.751 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:00 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:00.752 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:01.056 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T05:59:01.056 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T05:59:01.057 INFO:teuthology.run_tasks:Running task workunit... 2024-02-10T05:59:01.071 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:59:01.071 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-10T05:59:01.071 DEBUG:teuthology.orchestra.run.smithi121:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-10T05:59:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:59:01.087 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-10T05:59:01.087 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-10T05:59:01.142 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-10T05:59:01.142 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-10T05:59:01.197 INFO:tasks.workunit:timeout=3h 2024-02-10T05:59:01.197 INFO:tasks.workunit:cleanup=True 2024-02-10T05:59:01.198 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:59:01.252 INFO:tasks.workunit.client.0.smithi121.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-10T05:59:01.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2491404041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:59:01.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2491404041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:59:01.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2491404041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:59:02.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:02 smithi121 ceph-mon[41707]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:02.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:02 smithi121 ceph-mon[30020]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:02.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:02 smithi146 ceph-mon[33795]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:04 smithi121 ceph-mon[41707]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:04.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:04 smithi121 ceph-mon[30020]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:04.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:04 smithi146 ceph-mon[33795]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:59:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:59:06.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:06.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:06 smithi121 ceph-mon[41707]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:06.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:06 smithi121 ceph-mon[30020]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:06.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:06 smithi146 ceph-mon[33795]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:07.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:07.827 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:08.330 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:08 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[79124]: ts=2024-02-10T05:59:08.079Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003303983s 2024-02-10T05:59:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:08 smithi146 ceph-mon[33795]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:08.827 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:08 smithi121 ceph-mon[41707]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:08 smithi121 ceph-mon[30020]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:10 smithi146 ceph-mon[33795]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:10 smithi121 ceph-mon[30020]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:10 smithi121 ceph-mon[41707]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:11.671 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:11 smithi146 systemd[1]: Starting Ceph grafana.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:12.044 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:11 smithi146 podman[67183]: 2024-02-10 05:59:11.672722909 +0000 UTC m=+0.017527231 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-10T05:59:12.044 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:11 smithi146 podman[67183]: 2024-02-10 05:59:11.808399248 +0000 UTC m=+0.153203605 container create 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.buildah.version=1.32.0, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64) 2024-02-10T05:59:12.044 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 podman[67183]: 2024-02-10 05:59:12.043960693 +0000 UTC m=+0.388765027 container init 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.openshift.tags=base rhel8, name=ubi8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-02-10T05:59:12.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 podman[67183]: 2024-02-10 05:59:12.049655631 +0000 UTC m=+0.394459945 container start 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, architecture=x86_64, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, release=239.1651231664, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5) 2024-02-10T05:59:12.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 bash[67183]: 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c 2024-02-10T05:59:12.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 systemd[1]: Started Ceph grafana.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:12.296 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202243469Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202485345Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202509176Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202521838Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202533156Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202544126Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202554487Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202564826Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202575306Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202585997Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202596176Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202606305Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202616535Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-10T05:59:12.297 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=settings t=2024-02-10T05:59:12.202628258Z level=info msg="App mode production" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore t=2024-02-10T05:59:12.202695268Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore t=2024-02-10T05:59:12.202720052Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.20357976Z level=info msg="Starting DB migrations" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.203670754Z level=info msg="Executing migration" id="create migration_log table" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.204197635Z level=info msg="Executing migration" id="create user table" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.204621549Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.207347401Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.21013425Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.21279815Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.215295977Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.21799254Z level=info msg="Executing migration" id="create user table v2" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.218389861Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-10T05:59:12.298 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.221070112Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-10T05:59:12.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.223811354Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-10T05:59:12.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.226366238Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-10T05:59:12.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.229142122Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-10T05:59:12.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.231729905Z level=info msg="Executing migration" id="Update user table charset" 2024-02-10T05:59:12.299 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.234436392Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.234892924Z level=info msg="Executing migration" id="Add missing user data" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.237617926Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.240346169Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.241086692Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.241544209Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.242697773Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.24365736Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.244310812Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.245121352Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.246053242Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.246683802Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.247431059Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-10T05:59:12.300 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.247824094Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.248313367Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.24925482Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.249943744Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.25088158Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.251522018Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.252285198Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.252722777Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.253110841Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.253535004Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.253956575Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.254306719Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.25488962Z level=info msg="Executing migration" id="create star table" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.255425268Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.255846457Z level=info msg="Executing migration" id="create org table v1" 2024-02-10T05:59:12.301 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.256680787Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.257460984Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.257867404Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.258724888Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.259612712Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.260390233Z level=info msg="Executing migration" id="Update org table charset" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.260713903Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.261640358Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.262011863Z level=info msg="Executing migration" id="create dashboard table" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.262421762Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.263303215Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.263741148Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.266686562Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T05:59:12.302 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.26937502Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.272083798Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.273078475Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.275864164Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.278705775Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.281451916Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.281919879Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.284792241Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.287583241Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.290339177Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.290887504Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.293671074Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-10T05:59:12.303 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.296458083Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-10T05:59:12.547 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-mon[33795]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:12.548 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.548 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.548 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.548 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.299106332Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.2995651Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.302467223Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.302803155Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.305257996Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.308096764Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.310649221Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.313087923Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.315957488Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.318554672Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.319022045Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.321795716Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-10T05:59:12.549 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.32453167Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.326899181Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.327331287Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.330078974Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.332551141Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.335146525Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.335641752Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.338520498Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.339051894Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.339583656Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.34036351Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.340695005Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.341007788Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-10T05:59:12.550 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.341456376Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.341972378Z level=info msg="Executing migration" id="create data_source table" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.342427183Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.34286619Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.343327386Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.344138638Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.345097292Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.346287713Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.346721346Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.347598443Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.348329672Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.349190424Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.349761864Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.35062376Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.351609779Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-10T05:59:12.551 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.352218098Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.352808423Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.355628531Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.358427875Z level=info msg="Executing migration" id="Add uid column" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.361131117Z level=info msg="Executing migration" id="Update uid value" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.363950191Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.366491621Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.369134436Z level=info msg="Executing migration" id="create api_key table" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.372080358Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.372520753Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.375191796Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.377970448Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.380757308Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-10T05:59:12.552 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.383481333Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T05:59:12.553 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.386130846Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-10T05:59:12.553 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.388974275Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-10T05:59:12.553 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.391593382Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.394602185Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.397239973Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.399958693Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.400336757Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.402548649Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.402928129Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-10T05:59:12.554 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.405201059Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.407847769Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.410399522Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.41303252Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.415656119Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.418325518Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.420857239Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.423730095Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.426591281Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.429129988Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.431605395Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.434148745Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.436958948Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.439527296Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-10T05:59:12.555 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.442416739Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.443114169Z level=info msg="Executing migration" id="create quota table v1" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.443939613Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.444829665Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.445576374Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.446004615Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.446792919Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.447561908Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.447882803Z level=info msg="Executing migration" id="create session table" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.448356348Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.449100074Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.449434352Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-10T05:59:12.556 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.449876971Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.450686052Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.45144172Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.451760696Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.452040451Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.452347144Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.452742725Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.453026071Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.453664294Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.453972124Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.454659435Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.455612775Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.456277165Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.456730014Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-10T05:59:12.557 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.457661859Z level=info msg="Executing migration" id="create alert table v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.458364283Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.459186931Z level=info msg="Executing migration" id="add index alert state" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.459911977Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.460488145Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.461082884Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.46151361Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.46191586Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.46333139Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.463733396Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.464184719Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.464524993Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.464885899Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-10T05:59:12.558 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.465287191Z level=info msg="Executing migration" id="Add column is_default" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.465909334Z level=info msg="Executing migration" id="Add column frequency" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.466542377Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.467182439Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.467803587Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.468268604Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.468576744Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.46884841Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.469278448Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.469759489Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.470163826Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.47055929Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.470969922Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.471610058Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-10T05:59:12.559 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.472278814Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.472821296Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.47328673Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.473700366Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.474350072Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.474655891Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.475058286Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.475485675Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.475792556Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.476235917Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.477031385Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.477478317Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-10T05:59:12.560 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.477926895Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.478429719Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.479269622Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.48007976Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.480765524Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.481753683Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.482648369Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.483514645Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.484335999Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.485178572Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.486778289Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.487611719Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.488502104Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.489463052Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.490338545Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-10T05:59:12.561 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.491081456Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.491929329Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.492718386Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.493576534Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.496439198Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.499196972Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.500021082Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.500819147Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.501716466Z level=info msg="Executing migration" id="Move region to single row" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.502389342Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.503184037Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.504010546Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.50482405Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.505605674Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.506426212Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.507321867Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-10T05:59:12.562 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.508034835Z level=info msg="Executing migration" id="create test_data table" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.509044122Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.509666553Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.510512881Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.511321799Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.51210527Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.513047562Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.513923513Z level=info msg="Executing migration" id="create team table" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.514618371Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.515603886Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.516352916Z level=info msg="Executing migration" id="create team member table" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.517294158Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.518041064Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T05:59:12.563 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.519022728Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.519753391Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.520522773Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.521427128Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.522183999Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.523170542Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.5238947Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.524800162Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.525541699Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.52842426Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.531103815Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.533669837Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.534106794Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-10T05:59:12.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.536829741Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.539658774Z level=info msg="Executing migration" id="create tag table" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.542212524Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.542679671Z level=info msg="Executing migration" id="create login attempt table" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.545388565Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.548124677Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.550680962Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T05:59:12.565 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.552613441Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-10T05:59:12.799 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.555280845Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-10T05:59:12.799 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.558195091Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.560894083Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.561363131Z level=info msg="Executing migration" id="create user auth table" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.564113573Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.564631934Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.566817351Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.567602667Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.569391775Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.570156593Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.572738477Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.575496855Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.578015186Z level=info msg="Executing migration" id="create server_lock table" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.578453297Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-10T05:59:12.800 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.581105069Z level=info msg="Executing migration" id="create user auth token table" 2024-02-10T05:59:12.801 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.583730322Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-10T05:59:12.801 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.586312764Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-10T05:59:12.801 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.586735177Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-10T05:59:12.801 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.589641411Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-10T05:59:12.801 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.592299326Z level=info msg="Executing migration" id="create cache_data table" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.594725382Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.59516251Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.597646775Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.598573734Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.599276627Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.599598827Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.600445232Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.601356014Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.602059962Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.602346468Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.603110182Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.603546929Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-10T05:59:12.802 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.603935111Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.60437833Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.605153265Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.605591702Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.606420439Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.607085662Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.607529531Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.608314991Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.608625965Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.609016598Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.609452975Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.609836287Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.610257471Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.611093564Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T05:59:12.803 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.61189142Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.612328812Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.615757625Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.619120155Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.61995344Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.620379398Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.621245395Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.622103733Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.622950334Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.623426234Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.624270375Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.625080024Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.627829925Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-10T05:59:12.804 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.628684183Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.631451141Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.634164896Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.636838453Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.639525356Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.642306887Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.644935204Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.647620151Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.64848204Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.650966518Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.653854954Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.656450571Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.656917513Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T05:59:12.805 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.659549555Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-10T05:59:12.806 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.662390344Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.665154298Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.666050489Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.668815277Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.671615533Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.674291774Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.674679833Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.67734521Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.679934284Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.682711845Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.683306453Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.685894898Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-10T05:59:12.807 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.688762592Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.691145955Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.692013951Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.694883313Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.697592681Z level=info msg="Executing migration" id="create alert_image table" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.700131577Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.700563662Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.700854525Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.703250637Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.70610698Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.706184895Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.708784508Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.7092255Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.71215232Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-10T05:59:12.808 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.712268628Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.712378254Z level=info msg="alerts found to migrate" alerts=0 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.71470077Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.717533068Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.718002033Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.720658463Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.723227339Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.725823183Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.726155884Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.728896359Z level=info msg="Executing migration" id="create data_keys table" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.731491079Z level=info msg="Executing migration" id="create secrets table" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.734245273Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.738664778Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-10T05:59:12.809 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.741440032Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.744147801Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.748393898Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.75265592Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.755458459Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.757961079Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.760612702Z level=info msg="Executing migration" id="create permission table" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.761076231Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.761901469Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.762825984Z level=info msg="Executing migration" id="create role table" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.763506375Z level=info msg="Executing migration" id="add column display_name" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.764455359Z level=info msg="Executing migration" id="add column group_name" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.76547577Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.76626915Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-10T05:59:12.810 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.767190322Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.767695867Z level=info msg="Executing migration" id="create team role table" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.768629268Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.769302785Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.770134384Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.770580825Z level=info msg="Executing migration" id="create user role table" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.77155489Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.772194274Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.773036765Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.773469999Z level=info msg="Executing migration" id="create builtin role table" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.774273263Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.775183638Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.775851693Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.776823176Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.777343502Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T05:59:12.811 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.77806243Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.778504652Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.778922526Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.779328564Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.779789704Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.780755488Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.781200025Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.781664091Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.781957298Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.78238924Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.782845342Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.783243293Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.78356542Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.783910565Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.784328534Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-10T05:59:12.812 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.784795583Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.785759369Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.786072026Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.78650271Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.786964316Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.787424737Z level=info msg="Executing migration" id="add correlation config column" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.788470953Z level=info msg="Executing migration" id="create entity_events table" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.788932584Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.789725713Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.789830175Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.790187755Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.790282206Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.790564634Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-10T05:59:12.813 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.790928656Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.791364829Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.791960663Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.792404327Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.792793248Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.793220162Z level=info msg="Executing migration" id="Drop public config table" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.793606446Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.793999445Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.794424139Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.794842234Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.795287043Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.798314706Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.799404918Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.800983261Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-10T05:59:12.814 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.801417426Z level=info msg="Executing migration" id="add share column" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.803006286Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.80362241Z level=info msg="Executing migration" id="create file table" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.804233945Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.805004647Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.805659762Z level=info msg="Executing migration" id="create file_meta table" 2024-02-10T05:59:12.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.806249052Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-10T05:59:12.827 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[41707]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[30020]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.807087864Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.807880513Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.808763372Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.809073417Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.809533841Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-10T05:59:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.81060676Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.811592381Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.812239541Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.812654582Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.81552249Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.81593727Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.818299367Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.821042796Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.821491376Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.831217953Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.834151154Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.836718907Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-10T05:59:13.173 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.840323342Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.843131168Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.846003037Z level=info msg="Executing migration" id="create folder table" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.848679243Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.849180052Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.851919911Z level=info msg="Executing migration" id="Update folder title length" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.854486748Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=migrator t=2024-02-10T05:59:12.857336506Z level=info msg="migrations completed" performed=485 skipped=0 duration=653.690845ms 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore t=2024-02-10T05:59:12.857745041Z level=info msg="Created default organization" 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=plugin.loader t=2024-02-10T05:59:12.916803238Z level=info msg="Plugin registered" pluginID=input 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=plugin.loader t=2024-02-10T05:59:12.94404635Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-10T05:59:13.174 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=plugin.loader t=2024-02-10T05:59:12.944064473Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-10T05:59:13.175 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=secrets t=2024-02-10T05:59:12.9440985Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-10T05:59:13.175 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=query_data t=2024-02-10T05:59:12.946353294Z level=info msg="Query Service initialization" 2024-02-10T05:59:13.175 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=live.push_http t=2024-02-10T05:59:12.948706913Z level=info msg="Live Push Gateway initialization" 2024-02-10T05:59:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:13 smithi121 ceph-mon[41707]: Deploying daemon node-exporter.a on smithi121 2024-02-10T05:59:13.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:13 smithi121 ceph-mon[30020]: Deploying daemon node-exporter.a on smithi121 2024-02-10T05:59:13.593 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-mon[33795]: Deploying daemon node-exporter.a on smithi121 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=infra.usagestats.collector t=2024-02-10T05:59:13.56762667Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=provisioning.datasources t=2024-02-10T05:59:13.568580769Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=provisioning.datasources t=2024-02-10T05:59:13.57966543Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=provisioning.alerting t=2024-02-10T05:59:13.590355383Z level=info msg="starting to provision alerting" 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=provisioning.alerting t=2024-02-10T05:59:13.590373803Z level=info msg="finished to provision alerting" 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=ngalert.state.manager t=2024-02-10T05:59:13.590644239Z level=info msg="Warming state cache for startup" 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=ngalert.state.manager t=2024-02-10T05:59:13.590907124Z level=info msg="State cache has been initialized" states=0 duration=261.892µs 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=ticker t=2024-02-10T05:59:13.591097182Z level=info msg=starting first_tick=2024-02-10T05:59:20Z 2024-02-10T05:59:13.593 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=grafanaStorageLogger t=2024-02-10T05:59:13.590664786Z level=info msg="storage starting" 2024-02-10T05:59:13.594 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=ngalert.multiorg.alertmanager t=2024-02-10T05:59:13.591327806Z level=info msg="starting MultiOrg Alertmanager" 2024-02-10T05:59:13.594 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=http.server t=2024-02-10T05:59:13.593478913Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T05:59:13.921 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 05:59:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore.transactions t=2024-02-10T05:59:13.608177734Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T05:59:14.078 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:13 smithi121 bash[79698]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T05:59:14.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:14 smithi146 ceph-mon[33795]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:14 smithi121 ceph-mon[41707]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:14 smithi121 ceph-mon[30020]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:14.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:14 smithi121 bash[79698]: Getting image source signatures 2024-02-10T05:59:14.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:14 smithi121 bash[79698]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T05:59:14.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:14 smithi121 bash[79698]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T05:59:14.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:14 smithi121 bash[79698]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T05:59:15.327 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:59:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:59:16.315 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:15 smithi121 bash[79698]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T05:59:16.315 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 bash[79698]: Writing manifest to image destination 2024-02-10T05:59:16.315 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[41707]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:16.315 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:16.315 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:16.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[30020]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:16.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:16.316 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:16.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:16 smithi146 ceph-mon[33795]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:16.578 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 podman[79698]: 2024-02-10 05:59:16.294474317 +0000 UTC m=+2.604976966 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T05:59:16.578 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 podman[79698]: 2024-02-10 05:59:16.409997498 +0000 UTC m=+2.720500129 container create bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 podman[79698]: 2024-02-10 05:59:16.670722169 +0000 UTC m=+2.981224820 container init bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 podman[79698]: 2024-02-10 05:59:16.675729497 +0000 UTC m=+2.986232129 container start bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.679Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.679Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.679Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.679Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:59:17.006 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:59:17.007 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:59:17.008 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:59:17.009 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.680Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.681Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:59:17.010 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a[79748]: ts=2024-02-10T05:59:16.681Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:59:17.011 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 bash[79698]: bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 2024-02-10T05:59:17.011 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 05:59:16 smithi121 systemd[1]: Started Ceph node-exporter.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:17.819 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:17 smithi146 systemd[1]: Starting Ceph node-exporter.b for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:18 smithi146 ceph-mon[33795]: Deploying daemon node-exporter.b on smithi146 2024-02-10T05:59:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:18 smithi146 ceph-mon[33795]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:18.172 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:18 smithi146 bash[67471]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T05:59:18.278 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:18 smithi121 ceph-mon[41707]: Deploying daemon node-exporter.b on smithi146 2024-02-10T05:59:18.278 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:18 smithi121 ceph-mon[41707]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:18.279 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:18 smithi121 ceph-mon[30020]: Deploying daemon node-exporter.b on smithi146 2024-02-10T05:59:18.279 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:18 smithi121 ceph-mon[30020]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:18.921 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:18 smithi146 bash[67471]: Getting image source signatures 2024-02-10T05:59:18.921 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:18 smithi146 bash[67471]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T05:59:18.922 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:18 smithi146 bash[67471]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T05:59:18.922 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:18 smithi146 bash[67471]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T05:59:20.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:20 smithi121 ceph-mon[41707]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:20 smithi121 ceph-mon[30020]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:20.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:20 smithi146 ceph-mon[33795]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:21.171 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:20 smithi146 bash[67471]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T05:59:21.171 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:20 smithi146 bash[67471]: Writing manifest to image destination 2024-02-10T05:59:21.559 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 podman[67471]: 2024-02-10 05:59:21.249073986 +0000 UTC m=+3.326349807 container create dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:59:21.559 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 podman[67471]: 2024-02-10 05:59:21.167332687 +0000 UTC m=+3.244608514 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T05:59:21.559 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 podman[67471]: 2024-02-10 05:59:21.484581232 +0000 UTC m=+3.561857052 container init dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 podman[67471]: 2024-02-10 05:59:21.490486167 +0000 UTC m=+3.567761989 container start dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.494Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.494Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:59:21.560 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:59:21.561 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:59:21.562 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.495Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:59:21.563 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b[67521]: ts=2024-02-10T05:59:21.496Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:59:21.564 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 bash[67471]: dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 2024-02-10T05:59:21.921 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 05:59:21 smithi146 systemd[1]: Started Ceph node-exporter.b for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:22.900 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.902 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.902 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:22 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:22 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:24.875 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:24 smithi121 ceph-mon[30020]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:24.875 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:24 smithi121 ceph-mon[41707]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:24.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:24 smithi146 ceph-mon[33795]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:59:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[41707]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[30020]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:26 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:26 smithi146 ceph-mon[33795]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:26 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:26 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:26 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:26.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:25 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:27.327 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:27.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:27.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:28 smithi121 ceph-mon[41707]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:28 smithi121 ceph-mon[30020]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:28.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:28 smithi146 ceph-mon[33795]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:30 smithi121 ceph-mon[41707]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:30 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:30 smithi121 ceph-mon[30020]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:30 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:30.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:30 smithi146 ceph-mon[33795]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:30.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:30 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:32.751 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:32.752 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:33.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:33 smithi146 ceph-mon[33795]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T05:59:33.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:33 smithi146 ceph-mon[33795]: Reconfiguring daemon alertmanager.a on smithi121 2024-02-10T05:59:33.719 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:33 smithi121 ceph-mon[41707]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T05:59:33.719 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:33 smithi121 ceph-mon[41707]: Reconfiguring daemon alertmanager.a on smithi121 2024-02-10T05:59:33.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:33 smithi121 ceph-mon[30020]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T05:59:33.719 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:33 smithi121 ceph-mon[30020]: Reconfiguring daemon alertmanager.a on smithi121 2024-02-10T05:59:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:34 smithi146 ceph-mon[33795]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:34.675 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:34 smithi121 ceph-mon[41707]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:34.675 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:34 smithi121 ceph-mon[30020]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:35.294 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:59:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:59:36.078 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:35 smithi121 systemd[1]: Stopping Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:36.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:35 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:36.417 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[79124]: ts=2024-02-10T05:59:36.136Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:59:36.417 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 podman[80570]: 2024-02-10 05:59:36.157660185 +0000 UTC m=+0.154374211 container died c1c2d4d0986ec15a53f6f09bb15bb849bd1da5d28a26740630c64f82bfce78d6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:36.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:36 smithi146 ceph-mon[33795]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:36.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:36 smithi121 ceph-mon[41707]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:36.808 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:36 smithi121 ceph-mon[30020]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:36.809 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 podman[80570]: 2024-02-10 05:59:36.615289189 +0000 UTC m=+0.612003209 container cleanup c1c2d4d0986ec15a53f6f09bb15bb849bd1da5d28a26740630c64f82bfce78d6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:36.809 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 bash[80570]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T05:59:36.809 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 podman[80588]: 2024-02-10 05:59:36.808209684 +0000 UTC m=+0.644524284 container remove c1c2d4d0986ec15a53f6f09bb15bb849bd1da5d28a26740630c64f82bfce78d6 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:37.078 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:36 smithi121 podman[80588]: 2024-02-10 05:59:36.900825191 +0000 UTC m=+0.737139810 volume remove 685582c9af285f2e2d84cb290b66187d9b595c1849ccd99e37f297f6ae1cda6b 2024-02-10T05:59:37.549 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:37.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:37.550 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T05:59:37.550 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 systemd[1]: Stopped Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:37.550 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.068s CPU time. 2024-02-10T05:59:37.550 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 systemd[1]: Starting Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:37.828 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 podman[80679]: 2024-02-10 05:59:37.454393675 +0000 UTC m=+0.028306820 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-10T05:59:37.828 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 podman[80679]: 2024-02-10 05:59:37.576916018 +0000 UTC m=+0.150829154 volume create f6868f902e648f24f58e86ecfe9780dca0daf59df668d9aad94f1ba538421952 2024-02-10T05:59:37.828 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 podman[80679]: 2024-02-10 05:59:37.660596466 +0000 UTC m=+0.234509602 container create ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:38.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 podman[80679]: 2024-02-10 05:59:37.979915415 +0000 UTC m=+0.553828545 container init ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:38.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:37 smithi121 podman[80679]: 2024-02-10 05:59:37.9852992 +0000 UTC m=+0.559212337 container start ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:38.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.002Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T05:59:38.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.002Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T05:59:38.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.004Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.121 port=9094 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.006Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.045Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.046Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.050Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:38.050Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 bash[80679]: ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf 2024-02-10T05:59:38.329 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:38 smithi121 systemd[1]: Started Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[41707]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[30020]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:38 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:38 smithi146 ceph-mon[33795]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:38 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:38 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:39.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:39 smithi146 ceph-mon[33795]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T05:59:39.588 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:39 smithi146 ceph-mon[33795]: Reconfiguring daemon prometheus.a on smithi146 2024-02-10T05:59:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:39 smithi121 ceph-mon[41707]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T05:59:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:39 smithi121 ceph-mon[41707]: Reconfiguring daemon prometheus.a on smithi146 2024-02-10T05:59:39.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:39 smithi121 ceph-mon[30020]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T05:59:39.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:39 smithi121 ceph-mon[30020]: Reconfiguring daemon prometheus.a on smithi146 2024-02-10T05:59:40.328 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:40 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:40.006Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000537841s 2024-02-10T05:59:40.533 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:40 smithi146 ceph-mon[33795]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:40 smithi121 ceph-mon[41707]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:40 smithi121 ceph-mon[30020]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T05:59:42.171 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:41 smithi146 systemd[1]: Stopping Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.171Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T05:59:42.436 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T05:59:42.437 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T05:59:42.437 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T05:59:42.437 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T05:59:42.437 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[65182]: ts=2024-02-10T05:59:42.172Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T05:59:42.437 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 podman[68563]: 2024-02-10 05:59:42.182300431 +0000 UTC m=+0.130873693 container died c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:42.810 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:42 smithi146 ceph-mon[33795]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:42.811 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 podman[68563]: 2024-02-10 05:59:42.543622851 +0000 UTC m=+0.492196112 container cleanup c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:42.811 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 bash[68563]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T05:59:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:42 smithi121 ceph-mon[41707]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:42 smithi121 ceph-mon[30020]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:43.164 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 podman[68581]: 2024-02-10 05:59:42.811209217 +0000 UTC m=+0.630387168 container remove c3d0ed0fdc8c8cc3e38066aa1dec77e28f1f1502c01105ec4f6235ad066db075 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:43.165 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T05:59:43.165 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 systemd[1]: Stopped Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:43.165 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.158s CPU time. 2024-02-10T05:59:43.165 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:42 smithi146 systemd[1]: Starting Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:59:43.421 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 podman[68681]: 2024-02-10 05:59:43.069786359 +0000 UTC m=+0.020791064 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-10T05:59:43.421 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 podman[68681]: 2024-02-10 05:59:43.170769172 +0000 UTC m=+0.121773865 container create da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:43.421 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 podman[68681]: 2024-02-10 05:59:43.397770992 +0000 UTC m=+0.348775677 container init da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:43.422 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 podman[68681]: 2024-02-10 05:59:43.403499636 +0000 UTC m=+0.354504322 container start da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:59:43.922 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.427Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-10T05:59:43.922 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.427Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.427Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi146 (none))" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.427Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.427Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.428Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.428Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.429Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.429Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.841µs 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-10T05:59:43.923 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.432Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.861µs wal_replay_duration=556.983µs wbl_replay_duration=113ns total_replay_duration=597.548µs 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.433Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.433Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.433Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.450Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.546839ms db_storage=751ns remote_storage=1.049µs web_handler=314ns query_engine=526ns scrape=386.706µs scrape_sd=71.022µs notify=12.626µs notify_sd=17.342µs rules=16.852778ms tracing=5.631µs 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.451Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T05:59:43.451Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 bash[68681]: da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 2024-02-10T05:59:43.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 05:59:43 smithi146 systemd[1]: Started Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPING 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPED 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTING 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Serving on http://:::9283 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTED 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPING 2024-02-10T05:59:43.950 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPED 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTING 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Serving on http://:::9283 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTED 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPING 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STOPPED 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTING 2024-02-10T05:59:43.951 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Serving on http://:::9283 2024-02-10T05:59:44.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:05:59:43] ENGINE Bus STARTED 2024-02-10T05:59:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.832 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:44.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi121.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T05:59:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:44.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:59:45.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:45 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:45.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:45 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:45.836 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:45 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:59:46.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:45 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:46 smithi121 ceph-mon[41707]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:46 smithi121 ceph-mon[30020]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:46.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:46 smithi146 ceph-mon[33795]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:47.562 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:47.854 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:48.296 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T05:59:48.010Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003608788s 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[41707]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[30020]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:48.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:48 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:48 smithi146 ceph-mon[33795]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:48 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:48 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:50 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:59:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:50 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T05:59:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:52 smithi121 ceph-mon[41707]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:52 smithi121 ceph-mon[30020]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:52 smithi146 ceph-mon[33795]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:54 smithi121 ceph-mon[41707]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:54 smithi121 ceph-mon[30020]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:54.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:54 smithi146 ceph-mon[33795]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:55.327 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 05:59:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:05:59:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:59:56.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 05:59:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T05:59:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:56 smithi121 ceph-mon[41707]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:56 smithi121 ceph-mon[30020]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:56 smithi146 ceph-mon[33795]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:59:57.827 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr:state without impacting any branches by switching back to a branch. 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.502 INFO:tasks.workunit.client.0.smithi121.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: git switch -c 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr:Or undo this operation with: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: git switch - 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-02-10T05:59:58.503 INFO:tasks.workunit.client.0.smithi121.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-10T05:59:58.506 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-10T05:59:58.527 INFO:tasks.workunit.client.0.smithi121.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-10T05:59:58.529 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T05:59:58.529 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-10T05:59:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 05:59:58 smithi121 ceph-mon[41707]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 05:59:58 smithi121 ceph-mon[30020]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 05:59:58 smithi146 ceph-mon[33795]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:59:59.328 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-10T05:59:59.389 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-10T05:59:59.430 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-10T05:59:59.431 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T05:59:59.431 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-10T05:59:59.472 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-10T05:59:59.475 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-02-10T05:59:59.475 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-10T05:59:59.531 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-10T05:59:59.532 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-10T05:59:59.532 DEBUG:teuthology.orchestra.run.smithi121:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-10T05:59:59.606 INFO:tasks.workunit.client.0.smithi121.stderr:+ parallel=1 2024-02-10T05:59:59.606 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' '' = --serial ']' 2024-02-10T05:59:59.606 INFO:tasks.workunit.client.0.smithi121.stderr:+ crimson=0 2024-02-10T05:59:59.606 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' '' = --crimson ']' 2024-02-10T05:59:59.606 INFO:tasks.workunit.client.0.smithi121.stderr:+ color= 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -t 1 ']' 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ GTEST_OUTPUT=xml:/home/ubuntu/cephtest/archive/gtest_xml_report 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ declare -A pids 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_aio 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_aio' 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_aio 2024-02-10T05:59:59.607 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.608 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_aio 2024-02-10T05:59:59.608 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81453 2024-02-10T05:59:59.608 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_aio on pid 81453' 2024-02-10T05:59:59.608 INFO:tasks.workunit.client.0.smithi121.stdout:test api_aio on pid 81453 2024-02-10T05:59:59.611 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81453 2024-02-10T05:59:59.611 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.611 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.611 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-10T05:59:59.611 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_aio_pp 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_aio_pp' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_aio_pp 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.612 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.613 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_aio_pp 2024-02-10T05:59:59.614 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81460 2024-02-10T05:59:59.614 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_aio_pp on pid 81460' 2024-02-10T05:59:59.614 INFO:tasks.workunit.client.0.smithi121.stdout:test api_aio_pp on pid 81460 2024-02-10T05:59:59.617 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81460 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_io 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_io' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_io 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.618 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.619 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.620 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_io 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81469 2024-02-10T05:59:59.621 INFO:tasks.workunit.client.0.smithi121.stdout:test api_io on pid 81469 2024-02-10T05:59:59.625 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_io on pid 81469' 2024-02-10T05:59:59.625 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81469 2024-02-10T05:59:59.625 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_io_pp 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_io_pp' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_io_pp 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.626 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.627 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.628 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.629 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_io_pp 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81480 2024-02-10T05:59:59.630 INFO:tasks.workunit.client.0.smithi121.stdout:test api_io_pp on pid 81480 2024-02-10T05:59:59.636 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_io_pp on pid 81480' 2024-02-10T05:59:59.636 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81480 2024-02-10T05:59:59.636 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.636 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.636 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_asio 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_asio' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_asio 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.637 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.638 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.639 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.640 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.641 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_asio 2024-02-10T05:59:59.642 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81495 2024-02-10T05:59:59.643 INFO:tasks.workunit.client.0.smithi121.stdout:test api_asio on pid 81495 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_asio on pid 81495' 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81495 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_list 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.650 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_list' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_list 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.651 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.652 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.653 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.654 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.655 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.656 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.657 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81453/exe 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_list 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81512 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_list on pid 81512' 2024-02-10T05:59:59.658 INFO:tasks.workunit.client.0.smithi121.stdout:test api_list on pid 81512 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81512 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_lock 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_lock' 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_lock 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.666 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.667 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.668 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.669 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.670 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio.xml 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-10T05:59:59.671 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.672 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_aio: /' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81460/exe 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.673 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.674 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.675 INFO:tasks.workunit.client.0.smithi121.stdout:test api_lock on pid 81533 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_lock 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81533 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_lock on pid 81533' 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81533 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-10T05:59:59.680 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_lock_pp 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_lock_pp' 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.681 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.682 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_aio_pp.xml 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.683 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.684 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81469/exe 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.685 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.686 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_lock_pp 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.687 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.688 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_io.log 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.689 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.690 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_io: /' 2024-02-10T05:59:59.691 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.691 INFO:tasks.workunit.client.0.smithi121.stdout:test api_lock_pp on pid 81559 2024-02-10T05:59:59.694 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_lock_pp 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81559 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_lock_pp on pid 81559' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81559 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_misc 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_misc' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.695 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.696 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_misc 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.697 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.698 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81480/exe 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.699 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.702 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.703 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81495/exe 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.704 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io_pp.xml 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_misc 2024-02-10T05:59:59.705 INFO:tasks.workunit.client.0.smithi121.stdout:test api_misc on pid 81582 2024-02-10T05:59:59.709 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.709 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.709 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81582 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_misc on pid 81582' 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81582 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.710 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_asio.xml 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.711 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.712 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_asio: /' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_misc_pp 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_misc_pp' 2024-02-10T05:59:59.713 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.714 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_misc_pp 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.715 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.716 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81512/exe 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.717 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.718 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.719 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_list.xml 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.720 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.721 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.721 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_list: /' 2024-02-10T05:59:59.721 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.721 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_list.log 2024-02-10T05:59:59.721 INFO:tasks.workunit.client.0.smithi121.stdout:test api_misc_pp on pid 81607 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_misc_pp 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81607 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_misc_pp on pid 81607' 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81607 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.725 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.726 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.727 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.728 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.729 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.730 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.731 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.732 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81582/exe 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_tier_pp 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.733 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.734 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_tier_pp' 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.735 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_tier_pp 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc.xml 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_misc: /' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.736 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.737 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_tier_pp 2024-02-10T05:59:59.738 INFO:tasks.workunit.client.0.smithi121.stdout:test api_tier_pp on pid 81644 2024-02-10T05:59:59.741 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81644 2024-02-10T05:59:59.741 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.741 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_tier_pp on pid 81644' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81644 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.742 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.743 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_pool 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.744 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_pool' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_pool 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.745 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81533/exe 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81607/exe 2024-02-10T05:59:59.746 INFO:tasks.workunit.client.0.smithi121.stdout:test api_pool on pid 81675 2024-02-10T05:59:59.749 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.749 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.749 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_pool 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81675 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_pool on pid 81675' 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.750 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81675 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.751 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.752 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_snapshots 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_snapshots' 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-10T05:59:59.753 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_misc_pp.xml 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81559/exe 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_lock: /' 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.755 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock_pp.xml 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_snapshots 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.756 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.757 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_lock.xml 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.758 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.759 INFO:tasks.workunit.client.0.smithi121.stdout:test api_snapshots on pid 81741 2024-02-10T05:59:59.762 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_snapshots 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81741 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_snapshots on pid 81741' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81741 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_snapshots_pp 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_snapshots_pp' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.763 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.764 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.765 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.766 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_snapshots_pp 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.767 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.768 INFO:tasks.workunit.client.0.smithi121.stdout:test api_snapshots_pp on pid 81765 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_snapshots_pp 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81765 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_snapshots_pp on pid 81765' 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81765 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_stat 2024-02-10T05:59:59.770 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_stat' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_stat 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.771 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.772 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.775 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_stat 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.776 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.777 INFO:tasks.workunit.client.0.smithi121.stdout:test api_stat on pid 81776 2024-02-10T05:59:59.782 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.782 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81776 2024-02-10T05:59:59.782 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_stat on pid 81776' 2024-02-10T05:59:59.782 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81776 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.783 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.784 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.785 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.786 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.787 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.788 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.789 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.790 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.791 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81644/exe 2024-02-10T05:59:59.792 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_stat_pp 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_stat_pp' 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.793 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.794 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.795 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.796 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.797 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_stat_pp 2024-02-10T05:59:59.798 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_tier_pp.xml 2024-02-10T05:59:59.799 INFO:tasks.workunit.client.0.smithi121.stdout:test api_stat_pp on pid 81847 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_stat_pp 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.805 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81847 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_stat_pp on pid 81847' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81847 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.806 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_watch_notify 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81675/exe 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_watch_notify' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.807 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81741/exe 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_pool: /' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.808 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.809 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.810 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_pool.xml 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.811 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.812 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81765/exe 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots.xml 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.813 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_watch_notify 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.814 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.815 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.816 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.817 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-10T05:59:59.818 INFO:tasks.workunit.client.0.smithi121.stdout:test api_watch_notify on pid 81913 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_watch_notify 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81913 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_watch_notify on pid 81913' 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81913 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_snapshots_pp.xml 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81776/exe 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_watch_notify_pp 2024-02-10T05:59:59.821 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_watch_notify_pp' 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_stat: /' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat.xml 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_watch_notify_pp 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.823 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.824 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_watch_notify_pp 2024-02-10T05:59:59.825 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81927 2024-02-10T05:59:59.826 INFO:tasks.workunit.client.0.smithi121.stdout:test api_watch_notify_pp on pid 81927 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_watch_notify_pp on pid 81927' 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81927 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-10T05:59:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.832 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_cmd 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_cmd' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_cmd 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.834 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.835 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.836 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81847/exe 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.837 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.838 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_stat_pp.xml 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.839 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81913/exe 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.840 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.841 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify.xml 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.842 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-10T05:59:59.843 INFO:tasks.workunit.client.0.smithi121.stdout:test api_cmd on pid 81975 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_cmd 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81975 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_cmd on pid 81975' 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81975 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_cmd_pp 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_cmd_pp' 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_cmd_pp 2024-02-10T05:59:59.845 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.846 INFO:tasks.workunit.client.0.smithi121.stdout:test api_cmd_pp on pid 81992 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_cmd_pp 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81992 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_cmd_pp on pid 81992' 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=81992 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.849 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.850 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.851 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_service 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_service' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.852 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_service 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.853 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.854 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.855 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.855 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_service 2024-02-10T05:59:59.855 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.855 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.855 INFO:tasks.workunit.client.0.smithi121.stdout:test api_service on pid 82036 2024-02-10T05:59:59.860 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82036 2024-02-10T05:59:59.860 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.860 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_service on pid 82036' 2024-02-10T05:59:59.860 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82036 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_service_pp 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.861 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_service_pp' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.862 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.863 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81927/exe 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.864 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.865 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_service_pp 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_watch_notify_pp.xml 2024-02-10T05:59:59.866 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.867 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.868 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.869 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.870 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81975/exe 2024-02-10T05:59:59.871 INFO:tasks.workunit.client.0.smithi121.stdout:test api_service_pp on pid 82078 2024-02-10T05:59:59.873 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_service_pp 2024-02-10T05:59:59.873 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82078 2024-02-10T05:59:59.873 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_service_pp on pid 82078' 2024-02-10T05:59:59.873 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82078 2024-02-10T05:59:59.873 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.874 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_c_write_operations 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.875 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_c_write_operations' 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_c_write_operations 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.876 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.877 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.878 INFO:tasks.workunit.client.0.smithi121.stdout:test api_c_write_operations on pid 82092 2024-02-10T05:59:59.883 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_cmd: /' 2024-02-10T05:59:59.884 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_c_write_operations 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82092 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_c_write_operations on pid 82092' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82092 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd.xml 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.886 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.887 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.888 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_c_read_operations 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_c_read_operations' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.889 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_c_read_operations 2024-02-10T05:59:59.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.891 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.892 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.893 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.894 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_c_read_operations 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.895 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82128 2024-02-10T05:59:59.896 INFO:tasks.workunit.client.0.smithi121.stdout:test api_c_read_operations on pid 82128 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_c_read_operations on pid 82128' 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82128 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/81992/exe 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s api_cls_remote_reads 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.898 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' api_cls_remote_reads' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.899 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo api_cls_remote_reads 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.900 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.901 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.902 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.903 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82036/exe 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cmd_pp.xml 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=api_cls_remote_reads 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.904 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.905 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.905 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82147 2024-02-10T05:59:59.905 INFO:tasks.workunit.client.0.smithi121.stdout:test api_cls_remote_reads on pid 82147 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test api_cls_remote_reads on pid 82147' 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82147 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.907 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_service: /' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service.xml 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s list_parallel 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' list_parallel' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_service.log 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.908 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.909 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.910 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo list_parallel 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82078/exe 2024-02-10T05:59:59.911 INFO:tasks.workunit.client.0.smithi121.stdout:test list_parallel on pid 82161 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=list_parallel 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82161 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test list_parallel on pid 82161' 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82161 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.913 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s open_pools_parallel 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' open_pools_parallel' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.914 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo open_pools_parallel 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.915 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_service_pp.xml 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.916 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.917 INFO:tasks.workunit.client.0.smithi121.stdout:test open_pools_parallel on pid 82171 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=open_pools_parallel 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82171 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test open_pools_parallel on pid 82171' 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82171 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.922 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.923 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.924 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.925 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.926 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s delete_pools_parallel 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' delete_pools_parallel' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.927 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.928 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.929 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo delete_pools_parallel 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.930 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.931 INFO:tasks.workunit.client.0.smithi121.stdout:test delete_pools_parallel on pid 82199 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.933 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=delete_pools_parallel 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82199 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test delete_pools_parallel on pid 82199' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82199 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.934 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.935 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s cls 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.936 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' cls' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.937 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82092/exe 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo cls 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82128/exe 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.938 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.939 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.940 INFO:tasks.workunit.client.0.smithi121.stdout:test cls on pid 82222 2024-02-10T05:59:59.943 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=cls 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82222 2024-02-10T05:59:59.944 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test cls on pid 82222' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82222 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.945 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_write_operations.xml 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.946 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.947 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_c_read_operations.xml 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s cmd 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' cmd' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.948 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-10T05:59:59.949 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.950 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.951 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo cmd 2024-02-10T05:59:59.952 INFO:tasks.workunit.client.0.smithi121.stdout:test cmd on pid 82251 2024-02-10T05:59:59.953 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=cmd 2024-02-10T05:59:59.953 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82251 2024-02-10T05:59:59.953 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test cmd on pid 82251' 2024-02-10T05:59:59.953 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82251 2024-02-10T05:59:59.953 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s handler_error 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.954 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' handler_error' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo handler_error 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.955 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.956 INFO:tasks.workunit.client.0.smithi121.stdout:test handler_error on pid 82264 2024-02-10T05:59:59.958 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=handler_error 2024-02-10T05:59:59.958 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82264 2024-02-10T05:59:59.958 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test handler_error on pid 82264' 2024-02-10T05:59:59.958 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82264 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.959 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.960 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.961 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s io 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.962 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' io' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.963 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo io 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.964 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.965 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.966 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.966 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.966 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.966 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.966 INFO:tasks.workunit.client.0.smithi121.stdout:test io on pid 82280 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=io 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82280 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test io on pid 82280' 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82280 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.972 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.973 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s ec_io 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' ec_io' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.974 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo ec_io 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.975 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.976 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.977 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.978 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.979 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.980 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82161/exe 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T05:59:59.981 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T05:59:59.982 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.983 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T05:59:59.983 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T05:59:59.983 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82199/exe 2024-02-10T05:59:59.983 INFO:tasks.workunit.client.0.smithi121.stdout:test ec_io on pid 82310 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=ec_io 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82310 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test ec_io on pid 82310' 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82310 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T05:59:59.987 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.988 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T05:59:59.989 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.990 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-delete_pools_parallel.xml 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ list_parallel: /' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T05:59:59.991 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s list 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' list' 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T05:59:59.992 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T05:59:59.993 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.994 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T05:59:59.995 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo list 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T05:59:59.996 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=list 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T05:59:59.997 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82338 2024-02-10T05:59:59.998 INFO:tasks.workunit.client.0.smithi121.stdout:test list on pid 82338 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test list on pid 82338' 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82338 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.001 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.002 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.003 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.004 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s ec_list 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' ec_list' 2024-02-10T06:00:00.005 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82264/exe 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.006 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo ec_list 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_handler_error 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=ec_list 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.008 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_handler_error.log 2024-02-10T06:00:00.009 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.009 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.009 INFO:tasks.workunit.client.0.smithi121.stdout:test ec_list on pid 82368 2024-02-10T06:00:00.015 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.015 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.015 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.015 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82368 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test ec_list on pid 82368' 2024-02-10T06:00:00.016 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82368 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ handler_error: /' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s misc 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82147/exe 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' misc' 2024-02-10T06:00:00.017 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.018 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.019 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.020 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.021 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.022 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82280/exe 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82171/exe 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-list_parallel.xml 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.023 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_io 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ io: /' 2024-02-10T06:00:00.024 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_io.log 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-open_pools_parallel.xml 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.025 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.026 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82222/exe 2024-02-10T06:00:00.027 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.028 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ cls: /' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_cls.log 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_cls 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_cls_remote_reads.xml 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.029 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.030 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.031 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.032 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo misc 2024-02-10T06:00:00.033 INFO:tasks.workunit.client.0.smithi121.stdout:test misc on pid 82454 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=misc 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82454 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test misc on pid 82454' 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82454 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.037 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s pool 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' pool' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.038 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.039 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.040 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo pool 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.041 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.042 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82251/exe 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.043 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ cmd: /' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_cmd.log 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_cmd 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.044 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.045 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.046 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.047 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.047 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.047 INFO:tasks.workunit.client.0.smithi121.stdout:test pool on pid 82552 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.051 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=pool 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82552 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test pool on pid 82552' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82552 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s read_operations 2024-02-10T06:00:00.052 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' read_operations' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.053 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82368/exe 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.054 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.055 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.056 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_ec_list 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ ec_list: /' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_ec_list.log 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.057 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.058 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.059 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.060 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo read_operations 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82338/exe 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.061 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=read_operations 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82663 2024-02-10T06:00:00.062 INFO:tasks.workunit.client.0.smithi121.stdout:test read_operations on pid 82663 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test read_operations on pid 82663' 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82663 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ list: /' 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.066 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_list.log 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_list 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s snapshots 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.067 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' snapshots' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.068 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.069 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82310/exe 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_ec_io 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.070 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.071 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ ec_io: /' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_ec_io.log 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo snapshots 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.072 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.073 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.074 INFO:tasks.workunit.client.0.smithi121.stdout:test snapshots on pid 82718 2024-02-10T06:00:00.077 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=snapshots 2024-02-10T06:00:00.077 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82718 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test snapshots on pid 82718' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82718 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s watch_notify 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' watch_notify' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo watch_notify 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.078 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.079 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.080 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82454/exe 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.081 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.082 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.083 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_misc.log 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_misc 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.084 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ misc: /' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.085 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.086 INFO:tasks.workunit.client.0.smithi121.stdout:test watch_notify on pid 82794 2024-02-10T06:00:00.089 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=watch_notify 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82794 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test watch_notify on pid 82794' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82794 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.090 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:++ printf %25s write_operations 2024-02-10T06:00:00.091 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+ r=' write_operations' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.092 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82552/exe 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ awk '{print $1}' 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.093 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.094 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.095 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ echo write_operations 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_pool.log 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.096 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ pool: /' 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_pool 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+ ff=write_operations 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82820 2024-02-10T06:00:00.097 INFO:tasks.workunit.client.0.smithi121.stdout:test write_operations on pid 82820 2024-02-10T06:00:00.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'test write_operations on pid 82820' 2024-02-10T06:00:00.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ pids[$f]=82820 2024-02-10T06:00:00.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ ret=0 2024-02-10T06:00:00.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-02-10T06:00:00.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 1 -eq 1 ']' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81644 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81644 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.109 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.110 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -f /etc/bashrc ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:+ . /etc/bashrc 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -z '' ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ BASHRCSOURCED=Y 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ shopt -q login_shell 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:+++ umask 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' 0022 -eq 0 ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ SHELL=/bin/bash 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-02-10T06:00:00.111 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/bash_completion.sh 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/cobbler.sh 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorgrep.sh 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.112 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.113 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.114 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.115 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82718/exe 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'grep=grep --color=auto' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'egrep=egrep --color=auto' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorls.sh 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.116 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' '!' -t 0 ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ return 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export S_COLORS=auto 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ S_COLORS=auto 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.117 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.118 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_snapshots.log 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ snapshots: /' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_snapshots 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.119 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.120 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82663/exe 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-02-10T06:00:00.121 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ /usr/libexec/grepconf.sh -c 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.122 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.123 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_read_operations 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_read_operations.log 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82794/exe 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-02-10T06:00:00.124 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/debuginfod.sh 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/gawk.sh 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/lang.sh 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/locale 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ read_operations: /' 2024-02-10T06:00:00.125 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ watch_notify: /' 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG_backup=en_US.UTF-8 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /etc/locale.conf ']' 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-02-10T06:00:00.126 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -x /usr/bin/sed ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_watch_notify 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:++++ export LANG=en_US.UTF-8 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:++++ LANG=en_US.UTF-8 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ LANG=en_US.UTF-8 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ unset LANG_backup config 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n '' ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' dumb = linux ']' 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.127 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/less.sh 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ for i in /etc/profile.d/*.sh 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ '[' '' ']' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ . /etc/profile.d/which2.sh 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++++ readlink /proc/82820/exe 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++++ basename /usr/bin/bash 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_declare='declare -f' 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ which_opt=-f 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export which_declare 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:+++ export -f which 2024-02-10T06:00:00.128 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset i 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:++ unset -f pathmunge 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ export PATH 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ unset rc 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph_test_neorados_write_operations 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ tee ceph_test_neorados_write_operations.log 2024-02-10T06:00:00.129 INFO:tasks.workunit.client.0.smithi121.stderr:+ sed 's/^/ write_operations: /' 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[30020]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1367926380' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[30020]: overall HEALTH_OK 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[41707]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1367926380' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T06:00:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:00 smithi121 ceph-mon[41707]: overall HEALTH_OK 2024-02-10T06:00:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:00 smithi146 ceph-mon[33795]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1367926380' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-10T06:00:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:00 smithi146 ceph-mon[33795]: overall HEALTH_OK 2024-02-10T06:00:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/950100183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:01 smithi121 ceph-mon[41707]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:00:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/950100183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:01 smithi121 ceph-mon[30020]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:00:02.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/950100183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:02.218 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:01 smithi146 ceph-mon[33795]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T06:00:03.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:03.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:03.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:03.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:06.397 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:08.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-10T06:00:08.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [----------] Global test environment set-up. 2024-02-10T06:00:08.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-10T06:00:08.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (3 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-02-10T06:00:08.288 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [----------] 12 tests from AsioRados (15 ms total) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [----------] Global test environment tear-down 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (8607 ms total) 2024-02-10T06:00:08.289 INFO:tasks.workunit.client.0.smithi121.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-10T06:00:08.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: pgmap v57: 228 pgs: 83 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:00:08.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1598114969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/483738070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2822020520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: pgmap v57: 228 pgs: 83 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:00:08.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1598114969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/483738070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2822020520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: pgmap v57: 228 pgs: 83 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:00:08.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1598114969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/483738070' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2822020520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:08.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: pgmap v59: 388 pgs: 83 creating+peering, 132 active+clean, 173 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1416676282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/434315776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: pgmap v61: 580 pgs: 83 creating+peering, 132 active+clean, 365 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: pgmap v62: 580 pgs: 383 creating+peering, 145 active+clean, 52 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T06:00:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3763462390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1641546423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1150349365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3128373569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3415258222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1123866885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2419188797' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:09.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: pgmap v59: 388 pgs: 83 creating+peering, 132 active+clean, 173 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:09.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1416676282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/434315776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: pgmap v61: 580 pgs: 83 creating+peering, 132 active+clean, 365 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: pgmap v62: 580 pgs: 383 creating+peering, 145 active+clean, 52 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:09.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: pgmap v59: 388 pgs: 83 creating+peering, 132 active+clean, 173 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2152907193' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi121-81521-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2992515964' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi121-81537-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1416676282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/434315776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: pgmap v61: 580 pgs: 83 creating+peering, 132 active+clean, 365 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: pgmap v62: 580 pgs: 383 creating+peering, 145 active+clean, 52 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: Health check failed: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi121-82684-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3763462390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1641546423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1150349365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3128373569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3415258222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1123866885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi121-82579-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi121-81576-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi121-81602-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi121-81552-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3763462390' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1641546423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1150349365' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3128373569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3415258222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1123866885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T06:00:09.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2419188797' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:09.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:09.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2419188797' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] Global test environment set-up. 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.ListObjects (1368 ms) 2024-02-10T06:00:09.751 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (30 ms) 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo1,foo2,foo3 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo1 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo2 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo3 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo1,foo4,foo5 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo4 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo5 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo1 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo6,foo7 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo7 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo6 2024-02-10T06:00:09.752 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo4 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo5 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns2:foo7 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns2:foo6 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo1 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo1 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo2 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo3 2024-02-10T06:00:09.753 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (71 ms) 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 1 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 10 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 13 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 7 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 14 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 0 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 15 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 11 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 5 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 8 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 6 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 3 0 2024-02-10T06:00:09.754 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 4 0 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 12 0 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 9 0 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 2 0 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (59 ms) 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: x cursor=MIN 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=1 cursor=11:02547ec2:::1:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=10 cursor=11:52ea6a34:::10:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=13 cursor=11:566253c9:::13:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=7 cursor=11:5c6b0b28:::7:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=14 cursor=11:62a1935d:::14:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=0 cursor=11:6cac518f:::0:head 2024-02-10T06:00:09.755 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=15 cursor=11:863748b0:::15:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=11 cursor=11:89d3ae78:::11:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=5 cursor=11:b29083e3:::5:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=8 cursor=11:bd63b0f1:::8:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=6 cursor=11:c4fdafeb:::6:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=3 cursor=11:cfc208b3:::3:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=4 cursor=11:d83876eb:::4:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=12 cursor=11:de5d7c5f:::12:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=9 cursor=11:e960b815:::9:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > oid=2 cursor=11:f905c69b:::2:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=1 cursor=11:02547ec2:::1:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-02-10T06:00:09.756 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=10 cursor=11:52ea6a34:::10:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=13 cursor=11:566253c9:::13:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:566253c9:::13:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=7 cursor=11:5c6b0b28:::7:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-02-10T06:00:09.757 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-02-10T06:00:09.819 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=14 cursor=11:62a1935d:::14:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=0 cursor=11:6cac518f:::0:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=15 cursor=11:863748b0:::15:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:863748b0:::15:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:863748b0:::15:head -> 15 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=11 cursor=11:89d3ae78:::11:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:89d3ae78:::11:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=5 cursor=11:b29083e3:::5:head 2024-02-10T06:00:09.820 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:b29083e3:::5:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:b29083e3:::5:head -> 5 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=8 cursor=11:bd63b0f1:::8:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=6 cursor=11:c4fdafeb:::6:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=3 cursor=11:cfc208b3:::3:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:cfc208b3:::3:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=4 cursor=11:d83876eb:::4:head 2024-02-10T06:00:09.821 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:d83876eb:::4:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:d83876eb:::4:head -> 4 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=12 cursor=11:de5d7c5f:::12:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=9 cursor=11:e960b815:::9:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:e960b815:::9:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : oid=2 cursor=11:f905c69b:::2:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:566253c9:::13:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:566253c9:::13:head expected=11:566253c9:::13:head 2024-02-10T06:00:09.822 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=13 expected=13 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:5c6b0b28:::7:head expected=11:5c6b0b28:::7:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=7 expected=7 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:cfc208b3:::3:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:cfc208b3:::3:head expected=11:cfc208b3:::3:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=3 expected=3 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:c4fdafeb:::6:head expected=11:c4fdafeb:::6:head 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2024-02-10T06:00:09.823 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=6 expected=6 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:bd63b0f1:::8:head expected=11:bd63b0f1:::8:head 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=8 expected=8 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:f905c69b:::2:head expected=11:f905c69b:::2:head 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=2 expected=2 2024-02-10T06:00:09.824 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:e960b815:::9:head 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:e960b815:::9:head expected=11:e960b815:::9:head 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=9 expected=9 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:de5d7c5f:::12:head expected=11:de5d7c5f:::12:head 2024-02-10T06:00:09.825 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=12 list_parallel: process_1_[82486]: starting. 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: creating pool ceph_test_rados_list_parallel.smithi121-82316 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: created object 0... 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: created object 25... 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: created object 49... 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: finishing. 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_1_[82486]: shutting down. 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: starting. 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: listing objects. 2024-02-10T06:00:10.959 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: listed object 0... 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: listed object 25... 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: saw 50 objects 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_2_[82487]: shutting down. 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: starting. 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: creating pool ceph_test_rados_list_parallel.smithi121-82316 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: created object 0... 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: created object 25... 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: created object 49... 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: finishing. 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_3_[83539]: shutting down. 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.960 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: starting. 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: listing objects. 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: listed object 0... 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: listed object 25... 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: saw 43 objects 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_4_[83542]: shutting down. 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: starting. 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: removed 25 objects... 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: removed half of the objects 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: removed 50 objects... 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: removed 50 objects 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_5_[83544]: shutting down. 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.961 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: starting. 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: creating pool ceph_test_rados_list_parallel.smithi121-82316 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: created object 0... 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: created object 25... 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: created object 49... 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: finishing. 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_6_[83599]: shutting down. 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: starting. 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: listing objects. 2024-02-10T06:00:10.962 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: listed object 0... 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: listed object 25... 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: listed object 50... 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: saw 56 objects 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_7_[83600]: shutting down. 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: starting. 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: added 25 objects... 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: added half of the objects 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: added 50 objects... 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: added 50 objects 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_8_[83601]: shutting down. 2024-02-10T06:00:10.963 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:10.964 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:11.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.082 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3071138179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:11.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ********************** handler_error: Running main() from gmock_main.cc 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [----------] Global test environment set-up. 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (11933 ms) 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [----------] 1 test from neocls_handler_error (11933 ms total) 2024-02-10T06:00:11.889 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: 2024-02-10T06:00:11.890 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [----------] Global test environment tear-down 2024-02-10T06:00:11.890 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [==========] 1 test from 1 test suite ran. (11940 ms total) 2024-02-10T06:00:11.890 INFO:tasks.workunit.client.0.smithi121.stdout: handler_error: [ PASSED ] 1 test. 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11886 ms) 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11886 ms total) 2024-02-10T06:00:11.896 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: 2024-02-10T06:00:11.897 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-02-10T06:00:11.897 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11887 ms total) 2024-02-10T06:00:11.897 INFO:tasks.workunit.client.0.smithi121.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-02-10T06:00:11.898 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (70 ms) 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (56 ms) 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (11924 ms) 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (12050 ms total) 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (12050 ms total) 2024-02-10T06:00:11.899 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: Running main() from gmock_main.cc 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (60 ms) 2024-02-10T06:00:11.958 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (88 ms) 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (11925 ms) 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148355+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi121-81712-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148433+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi121-81906-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148481+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148521+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148563+0000 mon.a [INF] from='client.? 172.21.15.121:0/3456510460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi121-82223-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi121-81919-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148650+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148692+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi121-81884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148774+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi121-81694-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.959 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148855+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82085-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.960 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.148942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82141-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:11.960 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.149021+0000 mon.a [INF] from='client.? 172.21.15.121:0/2643323159' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi121-81959-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.149080+0000 mon.a [INF] from='client.? 172.21.15.121:0/3071138********* 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: starting. 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: creating pool ceph_test_rados_list_parallel.smithi121-82316 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: created object 0... 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: created object 25... 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: created object 49... 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: finishing. 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_9_[83724]: shutting down. 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.090 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: starting. 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: listing objects. 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: listed object 0... 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: listed object 25... 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: listed object 50... 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: listed object 75... 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: saw 93 objects 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_10_[83725]: shutting down. 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: starting. 2024-02-10T06:00:12.091 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: removed 25 objects... 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: removed half of the objects 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: removed 50 objects... 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: removed 50 objects 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_13_[83728]: shutting down. 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: starting. 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: added 25 objects... 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: added half of the objects 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: added 50 objects... 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: added 50 objects 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_11_[83726]: shutting down. 2024-02-10T06:00:12.092 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: starting. 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: added 25 objects... 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: added half of the objects 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: added 50 objects... 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: added 50 objects 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_12_[83727]: shutting down. 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: starting. 2024-02-10T06:00:12.093 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: creating pool ceph_test_rados_list_parallel.smithi121-82316 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: created object 0... 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: created object 25... 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: created object 49... 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: finishing. 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_14_[83806]: shutting down. 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: starting. 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listing objects. 2024-02-10T06:00:12.094 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 0... 2024-02-10T06:00:12.403 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 2 api_c_read_operations: Running main() from gmock_main.cc 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (12 ms) 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (11 ms) 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (13 ms) 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-10T06:00:12.404 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (30 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (7 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (4 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (3 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-02-10T06:00:12.405 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (3 ms) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (9 ms) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (3 ms) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (135 ms total) 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-10T06:00:12.406 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12525 ms total) 2024-02-10T06:00:12.407 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-02-10T06:00:12.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: pgmap v64: 1252 pgs: 383 creating+peering, 145 active+clean, 724 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/38641378' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: pgmap v64: 1252 pgs: 383 creating+peering, 145 active+clean, 724 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/38641378' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: pgmap v64: 1252 pgs: 383 creating+peering, 145 active+clean, 724 unknown; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 710 B/s rd, 0 op/s 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/38641378' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout:179' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi121-81966-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.149144+0000 mon.a [INF] from='client.? 172.21.15.121:0/3715896981' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi121-81810-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.149206+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82382-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.149287+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:09.795778+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.077737+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.079994+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:13.874 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.080718+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.421534+0000 mon.a [INF] from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi121-82684-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.421632+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi121-82579-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.421751+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.729496+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.734460+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:10.752180+0000 mon.b [INF] from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.573720+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.573870+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.889661+0000 mon.c [INF] from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.952118+0000 mon.b [INF] from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.955949+0000 mon.b [INF] from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.096755+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.097472+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:13.875 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.097973+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.098505+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.099016+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.099512+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.099999+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.100496+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.101032+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.101520+0000 mon.b [INF] from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.411113+0000 mon.a [INF] from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:13.876 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.653829+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: pgmap v67: 1076 pgs: 80 creating+peering, 996 active+clean; 638 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 78 KiB/s wr, 401 op/s 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: 15.0 deep-scrub starts 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: 15.0 deep-scrub ok 2024-02-10T06:00:14.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3511965633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: 15.4 deep-scrub starts 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: 15.4 deep-scrub ok 2024-02-10T06:00:14.174 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:13 smithi146 ceph-mon[33795]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: pgmap v67: 1076 pgs: 80 creating+peering, 996 active+clean; 638 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 78 KiB/s wr, 401 op/s 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:14.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:00:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: 15.0 deep-scrub starts 2024-02-10T06:00:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: 15.0 deep-scrub ok 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3511965633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: 15.4 deep-scrub starts 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: 15.4 deep-scrub ok 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[41707]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: pgmap v67: 1076 pgs: 80 creating+peering, 996 active+clean; 638 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 78 KiB/s wr, 401 op/s 2024-02-10T06:00:14.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi121-82061-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:14.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: 15.0 deep-scrub starts 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: 15.0 deep-scrub ok 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3511965633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: 15.4 deep-scrub starts 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:14.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:14.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:14.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:14.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: 15.4 deep-scrub ok 2024-02-10T06:00:14.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:13 smithi121 ceph-mon[30020]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.654007+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi121-81576-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.654068+0000 mon.a [INF] from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.662027+0000 mon.b [INF] from='client.? 172.21.15.121:0/3511965633' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.915850+0000 mon.a [INF] from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]: dispatch 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.916231+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.916947+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:12.919058+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.110 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.157295+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:11.956500+0000 client.admin [INF] onexx 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.871942+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.878353+0000 mon.a [INF] from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.878421+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.878488+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.884353+0000 mon.b [INF] from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.884943+0000 mon.b [INF] from='client.? 172.21.15.121:0/2204159170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.111 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.894668+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:13.926318+0000 mon.c [INF] from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain api_io_pp: Running main() from gmock_main.cc 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: seed 81576 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-10T06:00:15.266 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (101 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (10 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (1 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-02-10T06:00:15.267 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (4 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (1 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-10T06:00:15.268 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (2 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (7 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (3 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (2 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (2 ms) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (164 ms total) 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-10T06:00:15.269 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2324 ms) 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (12 ms) 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-10T06:00:15.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-02-10T06:00:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: onexx 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:15.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: 15.3 deep-scrub starts 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: 15.3 deep-scrub ok 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: pgmap v70: 828 pgs: 208 unknown, 72 creating+peering, 548 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 75 KiB/s wr, 529 op/s 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: 15.6 deep-scrub starts 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: 15.6 deep-scrub ok 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:15.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2204159170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: onexx 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:15.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: 15.3 deep-scrub starts 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: 15.3 deep-scrub ok 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: pgmap v70: 828 pgs: 208 unknown, 72 creating+peering, 548 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 75 KiB/s wr, 529 op/s 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: 15.6 deep-scrub starts 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: 15.6 deep-scrub ok 2024-02-10T06:00:15.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2204159170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:15.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:15.336 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:15.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:15.403 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] 5... 2024-02-10T06:00:15.403 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 50... 2024-02-10T06:00:15.403 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 75... 2024-02-10T06:00:15.403 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 100... 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: listed object 125... 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: saw 150 objects 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_15_[83807]: shutting down. 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: starting. 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: added 25 objects... 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: added half of the objects 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: added 50 objects... 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: added 50 objects 2024-02-10T06:00:15.404 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: process_16_[83808]: shutting down. 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******************************* 2024-02-10T06:00:15.405 INFO:tasks.workunit.client.0.smithi121.stdout: list_parallel: ******* SUCCESS ********** 2024-02-10T06:00:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: onexx 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: 15.3 deep-scrub starts 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: 15.3 deep-scrub ok 2024-02-10T06:00:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: pgmap v70: 828 pgs: 208 unknown, 72 creating+peering, 548 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 75 KiB/s wr, 529 op/s 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: 15.6 deep-scrub starts 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: 15.6 deep-scrub ok 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1627603988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi121-82684-1"}]': finished 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi121-81552-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2204159170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:15.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:15.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:15.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:15.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:16.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.5 deep-scrub starts 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.5 deep-scrub ok 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.9 deep-scrub starts 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.9 deep-scrub ok 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.2 deep-scrub starts 2024-02-10T06:00:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.2 deep-scrub ok 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.1 deep-scrub starts 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.1 deep-scrub ok 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.8 deep-scrub starts 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: 15.8 deep-scrub ok 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: twoxx 2024-02-10T06:00:16.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2610680173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.426 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.5 deep-scrub starts 2024-02-10T06:00:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.5 deep-scrub ok 2024-02-10T06:00:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.9 deep-scrub starts 2024-02-10T06:00:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.9 deep-scrub ok 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.2 deep-scrub starts 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.2 deep-scrub ok 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.1 deep-scrub starts 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.1 deep-scrub ok 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.8 deep-scrub starts 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: 15.8 deep-scrub ok 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: twoxx 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2610680173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.5 deep-scrub starts 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.5 deep-scrub ok 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.9 deep-scrub starts 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.9 deep-scrub ok 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.2 deep-scrub starts 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.2 deep-scrub ok 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.1 deep-scrub starts 2024-02-10T06:00:16.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.1 deep-scrub ok 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.8 deep-scrub starts 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: 15.8 deep-scrub ok 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: twoxx 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:16.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2610680173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:16.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:16.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:16.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.584 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.585 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:16.586 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:16.587 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.588 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.588 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:16.588 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.588 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:16.588 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:17.479 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: pgmap v72: 956 pgs: 368 unknown, 72 creating+peering, 516 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 66 KiB/s wr, 467 op/s 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.480 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: pgmap v72: 956 pgs: 368 unknown, 72 creating+peering, 516 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 66 KiB/s wr, 467 op/s 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:17.481 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.482 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.483 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.485 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: pgmap v72: 956 pgs: 368 unknown, 72 creating+peering, 516 active+clean; 599 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 66 KiB/s wr, 467 op/s 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:17.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:17.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:18.147 INFO:tasks.workunit.client.0.smithi121.stdout:=osd"]}]: dispatch 2024-02-10T06:00:18.147 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.156516+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.147 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.391104+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:18.147 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.392157+0000 mon.b [INF] from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:18.147 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.394964+0000 mon.c [INF] from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.625488+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:14.859611+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi121-81521-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.143235+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149451+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi121-81552-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149549+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi121-82684-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149634+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149697+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149740+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149801+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149853+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi121-81537-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.149939+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi121-81694-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:18.148 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.150020+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.150094+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.150161+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.150209+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi121-81919-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.155039+0000 mon.c [INF] from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.155358+0000 mon.b [INF] from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.155613+0000 mon.c [INF] from='client.? 172.21.15.121:0/2937633116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.155941+0000 mon.c [INF] from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.156116+0000 mon.c [INF] from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.157051+0000 mon.b [INF] from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.159182+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.159428+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.160832+0000 mon.c [INF] from='client.? 172.21.15.121:0/4131223116' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:18.149 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.162483+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:18.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.164513+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", " open_pools_parallel: process_1_[82414]: starting. 2024-02-10T06:00:18.465 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: creating pool ceph_test_rados_open_pools_parallel.smithi121-82391 2024-02-10T06:00:18.465 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: created object 0... 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: created object 25... 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: created object 49... 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: finishing. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_1_[82414]: shutting down. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_2_[82415]: starting. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_2_[82415]: rados_pool_create. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_2_[82415]: rados_ioctx_create. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_2_[82415]: shutting down. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: ******************************* 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: starting. 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: creating pool ceph_test_rados_open_pools_parallel.smithi121-82391 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: created object 0... 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: created object 25... 2024-02-10T06:00:18.466 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: created object 49... 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: finishing. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_3_[83860]: shutting down. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: ******************************* 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_4_[83861]: starting. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_4_[83861]: rados_pool_create. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_4_[83861]: rados_ioctx_create. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: process_4_[83861]: shutting down. 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: ******************************* 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: ******************************* 2024-02-10T06:00:18.467 INFO:tasks.workunit.client.0.smithi121.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-10T06:00:18.511 INFO:tasks.workunit.client.0.smithi121.stdout:pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:18.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.164863+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]: dispatch 2024-02-10T06:00:18.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.165181+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:18.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.175296+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:18.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.175883+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.177394+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:16.177976+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:17.211588+0000 mon.a [INF] from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:17.211634+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:17.211678+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:17.211748+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:17.211825+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.147812+0000 mon.b [INF] from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.444991+0000 mon.c [INF] from='client.? 172.21.15.121:0/2968579772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.445682+0000 mon.b [INF] from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.445878+0000 mon.a [WRN] Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2968579772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.761 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:18.762 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2968579772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:18.763 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:00:18.764 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2476177147' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi121-82061-1"}]': finished 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-4", "overlaypool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi121-82579-1"}]': finished 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi121-81537-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2968579772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: Health check update: 19 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T06:00:18.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.458101+000 cls: Running main() from gmock_main.cc 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [----------] Global test environment set-up. 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [ OK ] NeoRadosCls.DNE (11947 ms) 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (7585 ms) 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [----------] 2 tests from NeoRadosCls (19532 ms total) 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [----------] Global test environment tear-down 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [==========] 2 tests from 1 test suite ran. (19532 ms total) 2024-02-10T06:00:19.480 INFO:tasks.workunit.client.0.smithi121.stdout: cls: [ PASSED ] 2 tests. 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: Running main() from gmock_main.cc 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [----------] Global test environment set-up. 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9802 ms) 2024-02-10T06:00:19.484 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (5614 ms) 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (4077 ms) 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19493 ms total) 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [----------] Global test environment tear-down 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19493 ms total) 2024-02-10T06:00:19.485 INFO:tasks.workunit.client.0.smithi121.stdout: cmd: [ PASSED ] 3 tests. 2024-02-10T06:00:19.487 INFO:tasks.workunit.client.0.smithi121.stdout:LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-10T06:00:19.487 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2024-02-10T06:00:19.487 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-10T06:00:19.487 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (7 ms) 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-10T06:00:19.488 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2402 ms total) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (19829 ms total) 2024-02-10T06:00:19.489 INFO:tasks.workunit.client.0.smithi121.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-10T06:00:19.495 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: starting. 2024-02-10T06:00:19.495 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: creating pool ceph_test_rados_delete_pools_parallel.smithi121-82312 2024-02-10T06:00:19.495 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: created object 0... 2024-02-10T06:00:19.495 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: created object 25... 2024-02-10T06:00:19.495 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: created object 49... 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: finishing. 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_1_[82385]: shutting down. 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_2_[82390]: starting. 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_2_[82390]: deleting pool ceph_test_rados_delete_pools_parallel.smithi121-82312 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_2_[82390]: shutting down. 2024-02-10T06:00:19.496 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******************************* 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: starting. 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: creating pool ceph_test_rados_delete_pools_parallel.smithi121-82312 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: created object 0... 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: created object 25... 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: created object 49... 2024-02-10T06:00:19.497 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: finishing. 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_3_[83868]: shutting down. 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******************************* 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: starting. 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: listing objects. 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: listed object 0... 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: listed object 25... 2024-02-10T06:00:19.498 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: saw 50 objects 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_5_[83870]: shutting down. 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******************************* 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_4_[83869]: starting. 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_4_[83869]: deleting pool ceph_test_rados_delete_pools_parallel.smithi121-82312 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: process_4_[83869]: shutting down. 2024-02-10T06:00:19.499 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******************************* 2024-02-10T06:00:19.500 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******************************* 2024-02-10T06:00:19.500 INFO:tasks.workunit.client.0.smithi121.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-10T06:00:19.523 INFO:tasks.workunit.client.0.smithi121.stdout:0 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]: dispatch 2024-02-10T06:00:19.523 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.458325+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:00:19.524 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.459934+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi121-81694-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:19.524 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.460146+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi121-82684-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:19.524 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.460243+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi121-81712-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:19.524 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.460331+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi121-81959-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:19.524 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.460441+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi121-81919-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:19.525 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.477883+0000 mon.b [INF] from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.525 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.147998+0000 client.admin [INF] threexx 2024-02-10T06:00:19.525 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:18.515241+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.525 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.258118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:19.525 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.258393+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.526 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.258734+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.526 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.461868+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:19.526 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.463674+0000 mon.a [INF] from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:19.526 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.463740+0000 mon.a [INF] from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]': finished 2024-02-10T06:00:19.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: pgmap v75: 876 pgs: 104 unknown, 261 creating+peering, 511 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 38 op/s 2024-02-10T06:00:19.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: threexx 2024-02-10T06:00:19.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:19.806 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]': finished 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2397474748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:19.807 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: pgmap v75: 876 pgs: 104 unknown, 261 creating+peering, 511 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 38 op/s 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: threexx 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.808 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]': finished 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2397474748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.809 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.810 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: pgmap v75: 876 pgs: 104 unknown, 261 creating+peering, 511 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 38 op/s 2024-02-10T06:00:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: threexx 2024-02-10T06:00:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1407986323' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi121-81576-22"}]': finished 2024-02-10T06:00:19.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-4-cache", "mode": "writeback"}]': finished 2024-02-10T06:00:19.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:19.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:19.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2397474748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:19.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: fourxx 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: fourxx 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2141539457' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: fourxx 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]: dispatch 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: Running main() from gmock_main.cc 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] Global test environment set-up. 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (88 ms) 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-10T06:00:21.587 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: no timeout :/ 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: no timeout :/ 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: no timeout :/ 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: no timeout :/ 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: no timeout :/ 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (1247 ms) 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (8 ms) 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-02-10T06:00:21.588 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-10T06:00:21.589 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.RmXattr (14 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] 14 tests from LibRadosIo (1399 ms total) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1134 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (6 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-02-10T06:00:21.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (10 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (6 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1188 ms total) 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [----------] Global test environment tear-down 2024-02-10T06:00:21.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [==========] 24 tests from 2 test suites ran. (21938 ms total) 2024-02-10T06:00:21.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_io: [ PASSED ] 24 tests. 2024-02-10T06:00:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: pgmap v78: 948 pgs: 240 unknown, 210 creating+peering, 498 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 42 op/s 2024-02-10T06:00:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: 15.7 deep-scrub starts 2024-02-10T06:00:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: 15.7 deep-scrub ok 2024-02-10T06:00:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]': finished 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1688095211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:22.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]': finished 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: pgmap v78: 948 pgs: 240 unknown, 210 creating+peering, 498 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 42 op/s 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: 15.7 deep-scrub starts 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: 15.7 deep-scrub ok 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]': finished 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1688095211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:22.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]': finished 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: pgmap v78: 948 pgs: 240 unknown, 210 creating+peering, 498 active+clean; 145 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 42 op/s 2024-02-10T06:00:22.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: 15.7 deep-scrub starts 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: 15.7 deep-scrub ok 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-4"}]': finished 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1688095211' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/64889135' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-4", "tierpool": "test-rados-api-smithi121-81810-4-cache"}]': finished 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi121-81552-16"}]': finished 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-82061-6", "pg_num": 4}]': finished 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:22.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:22.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.463821+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi121-81521-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.463920+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.464012+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi121-82579-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.470236+0000 mon.b [INF] from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.475803+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi121-82579-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.488472+0000 mon.c [INF] from='client.? 172.21.15.121:0/2397474748' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.489299+0000 mon.a [INF] from='client.? 172.21.15.121:0/658502098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi121-81552-16"}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.496994+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.499118+0000 mon.c [INF] from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: got: 2024-02-10T06:00:19.499925+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (10937 ms) 2024-02-10T06:00:22.841 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (23010 ms total) 2024-02-10T06:00:22.842 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: 2024-02-10T06:00:22.842 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-10T06:00:22.842 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (23010 ms total) 2024-02-10T06:00:22.842 INFO:tasks.workunit.client.0.smithi121.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: pgmap v81: 844 pgs: 128 creating+peering, 716 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4068265311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1318974528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: Running main() from gmock_main.cc 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [----------] Global test environment set-up. 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-02-10T06:00:23.459 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (13844 ms) 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (9578 ms) 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (23422 ms total) 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [----------] Global test environment tear-down 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (23422 ms total) 2024-02-10T06:00:23.460 INFO:tasks.workunit.client.0.smithi121.stdout: ec_io: [ PASSED ] 2 tests. 2024-02-10T06:00:23.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: pgmap v81: 844 pgs: 128 creating+peering, 716 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: pgmap v81: 844 pgs: 128 creating+peering, 716 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 8.7 KiB/s wr, 16 op/s 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:23.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2886477062' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4068265311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1318974528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4068265311' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1318974528' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:23.583 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:23.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:23.584 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: Running main() from gmock_main.cc 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [----------] Global test environment set-up. 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [----------] 3 tests from NeoradosList 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ OK ] NeoradosList.ListObjects (11864 ms) 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ OK ] NeoradosList.ListObjectsNS (7593 ms) 2024-02-10T06:00:23.743 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4253 ms) 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: [----------] 3 tests from NeoradosList (23711 ms total) 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: [----------] Global test environment tear-down 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: [==========] 3 tests from 1 test suite ran. (23711 ms total) 2024-02-10T06:00:23.744 INFO:tasks.workunit.client.0.smithi121.stdout: list: [ PASSED ] 3 tests. 2024-02-10T06:00:24.496 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: seed 81712 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (260 ms) 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-10T06:00:24.497 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (51 ms) 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1029 ms) 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1037 ms) 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (6 ms) 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-10T06:00:24.498 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (17 ms) 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (11 ms) 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5 ms) 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2417 ms total) 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1192 ms) 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-10T06:00:24.499 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (4 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1053 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1014 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (3 ms) 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-10T06:00:24.500 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-02-10T06:00:24.501 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3276 ms total) 2024-02-10T06:00:24.501 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: 2024-02-10T06:00:24.501 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-10T06:00:24.501 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (24789 ms total) 2024-02-10T06:00:24.501 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: Running main() from gmock_main.cc 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] Global test environment set-up. 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStat.Stat (320 ms) 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (7 ms) 2024-02-10T06:00:24.509 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStat.StatNS (11 ms) 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] 5 tests from LibRadosStat (341 ms total) 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1212 ms) 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-10T06:00:24.510 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (10 ms) 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (5 ms) 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1227 ms total) 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [----------] Global test environment tear-down 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (24726 ms total) 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat: [ PASSED ] 9 tests. 2024-02-10T06:00:24.511 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: seed 81959 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (13 ms) 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (11 ms) 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-10T06:00:24.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (2 ms) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (11 ms) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (14 ms) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (51 ms total) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1199 ms) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-02-10T06:00:24.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (4 ms) 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (21 ms) 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1225 ms total) 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (24722 ms total) 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: Running main() from gmock_main.cc 2024-02-10T06:00:24.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] Global test environment set-up. 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (1387 ms) 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3 ms) 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1007 ms) 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1003 ms) 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3 ms) 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-10T06:00:24.515 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (5 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] 8 tests from LibRadosLock (3414 ms total) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1193 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (7 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1074 ms) 2024-02-10T06:00:24.516 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1037 ms) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (3 ms) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (3 ms) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (4 ms) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3325 ms total) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [----------] Global test environment tear-down 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (24804 ms total) 2024-02-10T06:00:24.517 INFO:tasks.workunit.client.0.smithi121.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-10T06:00:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: pgmap v84: 784 pgs: 140 unknown, 96 creating+peering, 548 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2024-02-10T06:00:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: pool 'PoolQuotaPP_smithi121-81537-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: pgmap v84: 784 pgs: 140 unknown, 96 creating+peering, 548 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: pool 'PoolQuotaPP_smithi121-81537-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:00:24.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: pgmap v84: 784 pgs: 140 unknown, 96 creating+peering, 548 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.7 KiB/s wr, 3 op/s 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: pool 'PoolQuotaPP_smithi121-81537-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi121-82684-2"}]': finished 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "overlaypool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi121-81633-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi121-81521-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2310571571' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3991397905' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1063192745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2647555677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:24.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:25.327 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]': finished 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/835568370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/835568370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:25.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi121-81966-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi121-81919-7"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi121-81694-10"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi121-81959-7"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi121-81712-10"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-82061-6", "mode": "writeback"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2273730447' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/835568370' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]: dispatch 2024-02-10T06:00:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:00:25.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:25.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:26.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: Running main() from gmock_main.cc 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [----------] Global test environment set-up. 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolList (9722 ms) 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-02-10T06:00:26.553 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (5608 ms) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3042 ms) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4139 ms) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1906 ms) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2055 ms) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [----------] 6 tests from NeoRadosPools (26473 ms total) 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: 2024-02-10T06:00:26.554 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [----------] Global test environment tear-down 2024-02-10T06:00:26.555 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [==========] 6 tests from 1 test suite ran. (26473 ms total) 2024-02-10T06:00:26.555 INFO:tasks.workunit.client.0.smithi121.stdout: pool: [ PASSED ] 6 tests. 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: pgmap v87: 624 pgs: 140 unknown, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: pgmap v87: 624 pgs: 140 unknown, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:26.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:26.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: pgmap v87: 624 pgs: 140 unknown, 484 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:26.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi121-82579-2"}]': finished 2024-02-10T06:00:26.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1326089837' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:27.624 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-10T06:00:27.624 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-10T06:00:27.624 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-10T06:00:27.624 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1547 ms) 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (17 ms) 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1564 ms total) 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (82 ms) 2024-02-10T06:00:27.625 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3933 ms) 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (4 ms) 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178124373504 notify_id 399431958529 notifier_gid 25208 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (4 ms) 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 399431958530 notifier_gid 25208 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-10T06:00:27.626 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 399431958531 notifier_gid 25208 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (4 ms) 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178124373504 notify_id 399431958531 notifier_gid 25208 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178124373504 notify_id 399431958532 notifier_gid 25208 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (4 ms) 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 399431958533 notifier_gid 25208 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (4 ms) 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: trying... 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 399431958534 notifier_gid 25208 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: timed out 2024-02-10T06:00:27.627 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: flushing 2024-02-10T06:00:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:00:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3716523531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3716523531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:28.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/311320730' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi121-81633-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi121-82579-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3716523531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watc api_watch_notify: Running main() from gmock_main.cc 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify_test_cb 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (24 ms) 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-10T06:00:28.693 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94493480519920 err -107 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1023 ms) 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94493480521184 err -107 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1006 ms) 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350976 cookie 94493480521184 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (5 ms) 2024-02-10T06:00:28.694 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350976 cookie 94493480575776 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (4 ms) 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350977 cookie 94493480555104 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (4 ms) 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350978 cookie 94493480555104 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350978 cookie 94493480591488 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (7 ms) 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 347892350976 cookie 94493480557392 2024-02-10T06:00:28.695 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 352187318273 cookie 94493480557392 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3082 ms) 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94493480557392 err -107 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_cb from 14862 notify_id 373662154755 cookie 94493480557392 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (4016 ms) 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94493480557392 err -107 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1006 ms) 2024-02-10T06:00:28.696 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10177 ms total) 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: watch_notify_test_cb 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1132 ms) 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1132 ms total) 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (28894 ms total) 2024-02-10T06:00:28.697 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: pgmap v90: 648 pgs: 96 creating+peering, 96 unknown, 456 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.5 KiB/s wr, 4 op/s 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2164959214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: pgmap v90: 648 pgs: 96 creating+peering, 96 unknown, 456 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.5 KiB/s wr, 4 op/s 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2164959214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: pgmap v90: 648 pgs: 96 creating+peering, 96 unknown, 456 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.5 KiB/s wr, 4 op/s 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/402488364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi121-81884-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi121-81521-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2934368988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi121-82579-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi121-81966-12"}]': finished 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2164959214' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:29.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [----------] Global test environment set-up. 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5078 ms) 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-10T06:00:30.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ OK ] LibRadosService.RegisterLate (28 ms) 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: cluster: 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: id: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: health: HEALTH_WARN 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 3 pool(s) do not have an application enabled 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: services: 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: mgr: y(active, since 113s), standbys: x 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: data: 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: pools: 9 pools, 228 pgs 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: objects: 230 objects, 582 KiB 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: pgs: 5.702% pgs unknown 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 36.404% pgs not active 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 132 active+clean 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 83 creating+peering 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 13 unknown 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: io: 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: cluster: 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: id: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: health: HEALTH_WARN 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 30 pool(s) do not have an application enabled 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: services: 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: mgr: y(active, since 115s), standbys: x 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: data: 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: pools: 41 pools, 1252 pgs 2024-02-10T06:00:30.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: objects: 230 objects, 582 KiB 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: usage: 236 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: pgs: 57.827% pgs unknown 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 30.591% pgs not active 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 724 unknown 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 383 creating+peering 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 145 active+clean 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: io: 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: client: 710 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5586 ms) 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ OK ] LibRadosService.Status (20044 ms) 2024-02-10T06:00:30.609 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [----------] 4 tests from LibRadosService (30736 ms total) 2024-02-10T06:00:30.610 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: 2024-02-10T06:00:30.610 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [----------] Global test environment tear-down 2024-02-10T06:00:30.610 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30736 ms total) 2024-02-10T06:00:30.610 INFO:tasks.workunit.client.0.smithi121.stdout: api_service: [ PASSED ] 4 tests. 2024-02-10T06:00:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: pgmap v93: 688 pgs: 32 creating+peering, 232 unknown, 424 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 3 op/s 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/29053205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2292511620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:30.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:30.838 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: pgmap v93: 688 pgs: 32 creating+peering, 232 unknown, 424 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 3 op/s 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/29053205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2292511620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.839 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:30.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:30.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: pgmap v93: 688 pgs: 32 creating+peering, 232 unknown, 424 active+clean; 145 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 3 op/s 2024-02-10T06:00:30.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:30.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3240355091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/29053205' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2292511620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:30.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:32.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:32.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:00:32.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:32.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:00:32.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:32.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi121-81521-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi121-81906-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: pgmap v96: 624 pgs: 27 creating+activating, 27 creating+peering, 104 unknown, 466 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1536035788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: pgmap v96: 624 pgs: 27 creating+activating, 27 creating+peering, 104 unknown, 466 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1536035788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:33.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: pgmap v96: 624 pgs: 27 creating+activating, 27 creating+peering, 104 unknown, 466 active+clean; 145 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:00:33.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:33.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T06:00:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]: dispatch 2024-02-10T06:00:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1536035788' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:34.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-10T06:00:34.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-10T06:00:34.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-10T06:00:34.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5073 ms) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (3185 ms) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20046 ms) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: attempt 0 of 20 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (6048 ms) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (34352 ms total) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (34352 ms total) 2024-02-10T06:00:34.246 INFO:tasks.workunit.client.0.smithi121.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-10T06:00:35.252 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: Running main() from gmock_main.cc 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [----------] Global test environment set-up. 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (17197 ms) 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (8322 ms) 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (9698 ms) 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [----------] 3 tests from NeoradosECList (35217 ms total) 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [----------] Global test environment tear-down 2024-02-10T06:00:35.253 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (35217 ms total) 2024-02-10T06:00:35.254 INFO:tasks.workunit.client.0.smithi121.stdout: ec_list: [ PASSED ] 3 tests. 2024-02-10T06:00:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/715022000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2511707690' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/715022000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2511707690' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-7"}]': finished 2024-02-10T06:00:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T06:00:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/715022000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2511707690' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:36.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: pgmap v98: 688 pgs: 96 unknown, 1 active, 40 creating+activating, 32 creating+peering, 519 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3071283775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3548669223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: pgmap v98: 688 pgs: 96 unknown, 1 active, 40 creating+activating, 32 creating+peering, 519 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3071283775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3548669223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: pgmap v98: 688 pgs: 96 unknown, 1 active, 40 creating+activating, 32 creating+peering, 519 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3071283775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]: dispatch 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/420851204' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi121-82579-3"}]': finished 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi121-81521-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3548669223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:36.582 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: pgmap v101: 680 pgs: 224 unknown, 14 creating+activating, 442 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1709336827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: pgmap v101: 680 pgs: 224 unknown, 14 creating+activating, 442 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:00:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1709336827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: pgmap v101: 680 pgs: 224 unknown, 14 creating+activating, 442 active+clean; 145 MiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1709336827' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:38.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: pgmap v104: 648 pgs: 64 creating+peering, 96 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4183544392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: pgmap v104: 648 pgs: 64 creating+peering, 96 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4183544392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:00:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: pgmap v104: 648 pgs: 64 creating+peering, 96 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]: dispatch 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi121-81537-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/543320992' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-82218-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:39.082 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:39.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4183544392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:39.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:39.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T06:00:39.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:39.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]': finished 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]: dispatch 2024-02-10T06:00:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]': finished 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]': finished 2024-02-10T06:00:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]': finished 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4"}]': finished 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi121-81521-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]: dispatch 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]: dispatch 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-9", "mode": "writeback"}]': finished 2024-02-10T06:00:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi121-82061-4", "tierpool": "test-rados-api-smithi121-82061-6"}]': finished 2024-02-10T06:00:40.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2205346243' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T06:00:40.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:00:40.229 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: Running main() from gmock_main.cc 2024-02-10T06:00:40.229 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-02-10T06:00:40.229 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [----------] Global test environment set-up. 2024-02-10T06:00:40.229 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-02-10T06:00:40.229 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10628 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (5433 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (5425 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2901 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (4201 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4355 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7176 ms) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (40119 ms total) 2024-02-10T06:00:40.230 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: 2024-02-10T06:00:40.231 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [----------] Global test environment tear-down 2024-02-10T06:00:40.231 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (40119 ms total) 2024-02-10T06:00:40.231 INFO:tasks.workunit.client.0.smithi121.stdout: write_operations: [ PASSED ] 7 tests. 2024-02-10T06:00:41.477 INFO:tasks.workunit.client.0.smithi121.stdout:h_notify_pp: flushed 2024-02-10T06:00:41.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: trying... 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 412316860423 notifier_gid 25208 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: timed out 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: flushing 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: flushed 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3044 ms) 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-10T06:00:41.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: List watches 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify2 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 420906795016 notifier_gid 25208 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify2 done 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: watch_check 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: unwatch2 2024-02-10T06:00:41.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: flushing 2024-02-10T06:00:41.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: done 2024-02-10T06:00:41.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3013 ms) 2024-02-10T06:00:41.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-10T06:00:41.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: List watches 2024-02-10T06:00:41.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify2 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: handle_notify cookie 94178123272448 notify_id 433791696900 notifier_gid 25208 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: notify2 done 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: watch_check 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: unwatch2 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: flushing 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: done 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3020 ms) 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16131 ms total) 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: 2024-02-10T06:00:41.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-10T06:00:41.482 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (41649 ms total) 2024-02-10T06:00:41.482 INFO:tasks.workunit.client.0.smithi121.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-10T06:00:41.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: pgmap v107: 680 pgs: 64 creating+peering, 128 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T06:00:41.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/241097176' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: pgmap v107: 680 pgs: 64 creating+peering, 128 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:41.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:00:41.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:41.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/241097176' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:41.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:41.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: pgmap v107: 680 pgs: 64 creating+peering, 128 unknown, 488 active+clean; 145 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2881692589' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi121-81537-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi121-82061-6", "pool2": "test-rados-api-smithi121-82061-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:41.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T06:00:41.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]: dispatch 2024-02-10T06:00:41.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/241097176' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:41.833 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: pgmap v110: 612 pgs: 48 creating+peering, 46 unknown, 518 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/618672779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: pgmap v110: 612 pgs: 48 creating+peering, 46 unknown, 518 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/618672779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: pgmap v110: 612 pgs: 48 creating+peering, 46 unknown, 518 active+clean; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-9"}]': finished 2024-02-10T06:00:42.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:42.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T06:00:42.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/618672779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:42.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:43.464 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-02-10T06:00:43.464 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-10T06:00:43.464 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-10T06:00:43.464 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-10T06:00:43.464 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12006 ms) 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (7590 ms) 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4253 ms) 2024-02-10T06:00:43.465 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2814 ms) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4285 ms) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4436 ms) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8184 ms) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (43568 ms total) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (43571 ms total) 2024-02-10T06:00:43.466 INFO:tasks.workunit.client.0.smithi121.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-10T06:00:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1008671938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:43.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: 15.0 deep-scrub starts 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: 15.0 deep-scrub ok 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:43.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:43.840 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1008671938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:43.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: 15.0 deep-scrub starts 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: 15.0 deep-scrub ok 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:43.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:43.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi121-81521-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:43.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1008671938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: 15.0 deep-scrub starts 2024-02-10T06:00:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: 15.0 deep-scrub ok 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:43.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:45.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:45.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi121-81537-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:45.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:45.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T06:00:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:46.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:46.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:46.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:46.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:46.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: pgmap v113: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.4 deep-scrub starts 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.4 deep-scrub ok 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.6 deep-scrub starts 2024-02-10T06:00:46.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.6 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.3 deep-scrub starts 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.3 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.5 deep-scrub starts 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.5 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.2 deep-scrub starts 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.2 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.7 deep-scrub starts 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.7 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.9 deep-scrub starts 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.9 deep-scrub ok 2024-02-10T06:00:46.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.8 deep-scrub starts 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.8 deep-scrub ok 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: pgmap v116: 548 pgs: 160 unknown, 388 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2840145459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.1 deep-scrub starts 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: 15.1 deep-scrub ok 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:46.334 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:46.335 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: pgmap v113: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.4 deep-scrub starts 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.4 deep-scrub ok 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.6 deep-scrub starts 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.6 deep-scrub ok 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.3 deep-scrub starts 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.3 deep-scrub ok 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.5 deep-scrub starts 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.5 deep-scrub ok 2024-02-10T06:00:46.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.2 deep-scrub starts 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.2 deep-scrub ok 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.7 deep-scrub starts 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.7 deep-scrub ok 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.9 deep-scrub starts 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.9 deep-scrub ok 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.8 deep-scrub starts 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.8 deep-scrub ok 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: pgmap v116: 548 pgs: 160 unknown, 388 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:46.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2840145459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.1 deep-scrub starts 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: 15.1 deep-scrub ok 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:46.338 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:45 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:46.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-02-10T06:00:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: pgmap v113: 612 pgs: 32 creating+peering, 160 unknown, 420 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.4 deep-scrub starts 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.4 deep-scrub ok 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.6 deep-scrub starts 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.6 deep-scrub ok 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.3 deep-scrub starts 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.3 deep-scrub ok 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.5 deep-scrub starts 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.5 deep-scrub ok 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.2 deep-scrub starts 2024-02-10T06:00:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.2 deep-scrub ok 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.7 deep-scrub starts 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.7 deep-scrub ok 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.9 deep-scrub starts 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.9 deep-scrub ok 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.8 deep-scrub starts 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.8 deep-scrub ok 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: pgmap v116: 548 pgs: 160 unknown, 388 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi121-81884-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2840145459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.1 deep-scrub starts 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: 15.1 deep-scrub ok 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:45 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:00:47.728 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:47.728 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1549757488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:47.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1549757488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81870-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi121-81521-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1549757488' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:47.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:48.502 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-10T06:00:48.502 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] Global test environment set-up. 2024-02-10T06:00:48.502 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-10T06:00:48.502 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T05:59:59.751+0000 7f8b42248780 -1 failed for service _ceph-mon._tcp 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T05:59:59.751+0000 7f8b42248780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (52 ms) 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2010 ms) 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2062 ms total) 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-10T06:00:48.503 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: started 0x55a0528ae720 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: started 0x55a052bc1030 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: started 2 aios 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: waiting 0x55a0528ae720 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: waiting 0x55a052bc1030 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: done. 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (14395 ms) 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (14395 ms total) 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-10T06:00:48.504 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.Exec (205 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (16 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6840 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21642 ms) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28706 ms total) 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: 2024-02-10T06:00:48.505 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [----------] Global test environment tear-down 2024-02-10T06:00:48.506 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (48781 ms total) 2024-02-10T06:00:48.506 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: pgmap v119: 524 pgs: 72 creating+peering, 32 unknown, 420 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 18 MiB/s wr, 4 op/s 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]: dispatch 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T06:00:48.760 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: pgmap v119: 524 pgs: 72 creating+peering, 32 unknown, 420 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 18 MiB/s wr, 4 op/s 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]: dispatch 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T06:00:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: pgmap v119: 524 pgs: 72 creating+peering, 32 unknown, 420 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 18 MiB/s wr, 4 op/s 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi121-81537-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]: dispatch 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]: dispatch 2024-02-10T06:00:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]': finished 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]': finished 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1483121109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]: dispatch 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]': finished 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]': finished 2024-02-10T06:00:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1483121109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]: dispatch 2024-02-10T06:00:49.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-11", "mode": "writeback"}]': finished 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache", "force_nonempty":""}]': finished 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1483121109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81870-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi121-81521-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T06:00:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]: dispatch 2024-02-10T06:00:49.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: pgmap v122: 588 pgs: 40 creating+peering, 160 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]': finished 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:00:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: pgmap v122: 588 pgs: 40 creating+peering, 160 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]': finished 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:00:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: pgmap v122: 588 pgs: 40 creating+peering, 160 unknown, 388 active+clean; 217 MiB data, 1013 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-02-10T06:00:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:00:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2510849210' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81870-10", "tierpool":"test-rados-api-smithi121-81870-10-cache"}]': finished 2024-02-10T06:00:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2245475343' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:00:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T06:00:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]: dispatch 2024-02-10T06:00:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:00:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:00:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:00:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-11"}]': finished 2024-02-10T06:00:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T06:00:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: Running main() from gmock_main.cc 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [----------] Global test environment set-up. 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-02-10T06:00:52.234 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.Version (9433 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (5905 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.LongName (4086 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4245 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2834 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4278 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.Exec (4423 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3426 ms) 2024-02-10T06:00:52.235 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3742 ms) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3897 ms) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2877 ms) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3020 ms) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [----------] 12 tests from NeoRadosMisc (52166 ms total) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [----------] Global test environment tear-down 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [==========] 12 tests from 1 test suite ran. (52166 ms total) 2024-02-10T06:00:52.236 INFO:tasks.workunit.client.0.smithi121.stdout: misc: [ PASSED ] 12 tests. 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: Running main() from gmock_main.cc 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [----------] Global test environment set-up. 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolList (10971 ms) 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7724 ms) 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3131 ms) 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-10T06:00:52.282 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2922 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3446 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5102 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6161 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5398 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7648 ms) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [----------] 9 tests from LibRadosPools (52503 ms total) 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [----------] Global test environment tear-down 2024-02-10T06:00:52.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (52503 ms total) 2024-02-10T06:00:52.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-10T06:00:53.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: pgmap v125: 588 pgs: 74 creating+peering, 60 unknown, 454 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-02-10T06:00:53.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:53.314 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:53.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:53.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T06:00:53.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4131527122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:53.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: pgmap v125: 588 pgs: 74 creating+peering, 60 unknown, 454 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4131527122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: pgmap v125: 588 pgs: 74 creating+peering, 60 unknown, 454 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-02-10T06:00:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/402065609' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi121-81521-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T06:00:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4131527122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: pgmap v128: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:54.110 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:53 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: pgmap v128: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: pgmap v128: 428 pgs: 64 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi121-81537-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:53 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:00:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:00:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[41707]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T06:00:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[30020]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T06:00:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:55.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:55.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:55.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:54 smithi146 ceph-mon[33795]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T06:00:55.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: pgmap v131: 492 pgs: 128 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3524383750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: pgmap v131: 492 pgs: 128 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:00:56.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3524383750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:56.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:55 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: pgmap v131: 492 pgs: 128 unknown, 364 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1078793123' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi121-81521-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3524383750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:00:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:55 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:00:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:57.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:57.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:00:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi121-81537-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3687941483' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T06:00:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]: dispatch 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: Running main() from gmock_main.cc 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: [----------] Global test environment set-up. 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: [----------] 14 tests from NeoRadosIo 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-02-10T06:00:58.244 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.Limits (11950 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (4286 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.ReadOp (4659 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.SparseRead (2904 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2820 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4297 ms) 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-02-10T06:00:58.245 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (4425 ms) 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (4910 ms) 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3238 ms) 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.Trunc (2915 ms) 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.Remove (2868 ms) 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-02-10T06:00:58.246 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3015 ms) 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.RmXattr (3000 ms) 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3013 ms) 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [----------] 14 tests from NeoRadosIo (58300 ms total) 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [----------] Global test environment tear-down 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [==========] 14 tests from 1 test suite ran. (58305 ms total) 2024-02-10T06:00:58.247 INFO:tasks.workunit.client.0.smithi121.stdout: io: [ PASSED ] 14 tests. 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: Running main() from gmock_main.cc 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [----------] Global test environment set-up. 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (11803 ms) 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (7590 ms) 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-02-10T06:00:58.252 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3105 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2957 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (4311 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3201 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3436 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3720 ms) 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-02-10T06:00:58.253 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3235 ms) 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2883 ms) 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2886 ms) 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3028 ms) 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-02-10T06:00:58.254 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2976 ms) 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3029 ms) 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (58160 ms total) 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [----------] Global test environment tear-down 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (58160 ms total) 2024-02-10T06:00:58.255 INFO:tasks.workunit.client.0.smithi121.stdout: read_operations: [ PASSED ] 14 tests. 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: pgmap v134: 484 pgs: 32 creating+peering, 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/779516983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: pgmap v134: 484 pgs: 32 creating+peering, 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/779516983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:58.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: pgmap v134: 484 pgs: 32 creating+peering, 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:00:58.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi121-81884-10"}]': finished 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/779516983' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3844869835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3844869835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:00:59.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi121-81521-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3844869835' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:00:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: pgmap v137: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: pgmap v137: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:00.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T06:01:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:00:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:00.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: pgmap v137: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 305 KiB/s wr, 51 op/s 2024-02-10T06:01:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi121-81537-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T06:01:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:00:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:01:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4055929742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4055929742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:01.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:01.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi121-81884-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:01.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:01.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T06:01:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]: dispatch 2024-02-10T06:01:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4055929742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: pgmap v140: 428 pgs: 26 creating+peering, 14 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 KiB/s wr, 41 op/s 2024-02-10T06:01:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]': finished 2024-02-10T06:01:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2577985287' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: pgmap v140: 428 pgs: 26 creating+peering, 14 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 KiB/s wr, 41 op/s 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]': finished 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2577985287' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:02.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:02.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: pgmap v140: 428 pgs: 26 creating+peering, 14 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 KiB/s wr, 41 op/s 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-13", "mode": "writeback"}]': finished 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi121-81521-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2577985287' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:02.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.5 scrub starts 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.5 scrub ok 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.8 scrub starts 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.8 scrub ok 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.2 scrub starts 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.2 scrub ok 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T06:01:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.4 scrub starts 2024-02-10T06:01:03.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.4 scrub ok 2024-02-10T06:01:03.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.3 scrub starts 2024-02-10T06:01:03.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[41707]: 159.3 scrub ok 2024-02-10T06:01:03.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:03.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.5 scrub starts 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.5 scrub ok 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:03.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.8 scrub starts 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.8 scrub ok 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.2 scrub starts 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.2 scrub ok 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.4 scrub starts 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.4 scrub ok 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.3 scrub starts 2024-02-10T06:01:03.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:02 smithi121 ceph-mon[30020]: 159.3 scrub ok 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.0"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.1"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.2"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.3"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.4"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.5"}]: dispatch 2024-02-10T06:01:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.6"}]: dispatch 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.7"}]: dispatch 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.8"}]: dispatch 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "159.9"}]: dispatch 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.5 scrub starts 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.5 scrub ok 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.8 scrub starts 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.8 scrub ok 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.2 scrub starts 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.2 scrub ok 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi121-81537-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.4 scrub starts 2024-02-10T06:01:03.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.4 scrub ok 2024-02-10T06:01:03.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.3 scrub starts 2024-02-10T06:01:03.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:02 smithi146 ceph-mon[33795]: 159.3 scrub ok 2024-02-10T06:01:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: 159.7 scrub starts 2024-02-10T06:01:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: 159.7 scrub ok 2024-02-10T06:01:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: pgmap v143: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 KiB/s wr, 76 op/s 2024-02-10T06:01:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: 159.1 deep-scrub starts 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[41707]: 159.1 deep-scrub ok 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: 159.7 scrub starts 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: 159.7 scrub ok 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: pgmap v143: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 KiB/s wr, 76 op/s 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T06:01:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: 159.1 deep-scrub starts 2024-02-10T06:01:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:03 smithi121 ceph-mon[30020]: 159.1 deep-scrub ok 2024-02-10T06:01:04.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: 159.7 scrub starts 2024-02-10T06:01:04.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: 159.7 scrub ok 2024-02-10T06:01:04.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: pgmap v143: 428 pgs: 32 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 KiB/s wr, 76 op/s 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: 159.1 deep-scrub starts 2024-02-10T06:01:04.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:03 smithi146 ceph-mon[33795]: 159.1 deep-scrub ok 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: 159.6 scrub starts 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: 159.6 scrub ok 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: 159.6 scrub starts 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: 159.6 scrub ok 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T06:01:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:05.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: 159.6 scrub starts 2024-02-10T06:01:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: 159.6 scrub ok 2024-02-10T06:01:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/483538299' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi121-81521-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T06:01:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:06.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: 159.9 scrub starts 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: 159.9 scrub ok 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: pgmap v146: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s wr, 35 op/s 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:05 smithi146 ceph-mon[33795]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: 159.9 scrub starts 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: 159.9 scrub ok 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: pgmap v146: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s wr, 35 op/s 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[41707]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: 159.9 scrub starts 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: 159.9 scrub ok 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: pgmap v146: 460 pgs: 64 unknown, 8 creating+peering, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s wr, 35 op/s 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2266958872' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi121-81537-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:06.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:05 smithi121 ceph-mon[30020]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T06:01:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[41707]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2563566654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[30020]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2563566654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:07.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:07.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-mon[33795]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T06:01:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2563566654' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: pgmap v149: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1506466251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: pgmap v149: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1506466251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: pgmap v149: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi121-81521-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1506466251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:09 smithi121 ceph-mon[41707]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:09 smithi121 ceph-mon[41707]: 159.0 scrub starts 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:09 smithi121 ceph-mon[41707]: 159.0 scrub ok 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:08 smithi121 ceph-mon[30020]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:08 smithi121 ceph-mon[30020]: 159.0 scrub starts 2024-02-10T06:01:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:08 smithi121 ceph-mon[30020]: 159.0 scrub ok 2024-02-10T06:01:09.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi121-81537-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:08 smithi146 ceph-mon[33795]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T06:01:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:08 smithi146 ceph-mon[33795]: 159.0 scrub starts 2024-02-10T06:01:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:08 smithi146 ceph-mon[33795]: 159.0 scrub ok 2024-02-10T06:01:10.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: pgmap v152: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:10.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:10.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:10.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T06:01:10.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2812894430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: pgmap v152: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2812894430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:10.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:10.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: pgmap v152: 428 pgs: 32 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 6.0 KiB/s wr, 77 op/s 2024-02-10T06:01:10.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:10.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:10.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T06:01:10.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:10.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2812894430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:10.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (6242 ms) 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3045 ms) 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-10T06:01:11.233 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (4136 ms) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1908 ms) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (15331 ms total) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4123 ms) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4517 ms) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4957 ms) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13597 ms total) 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: 2024-02-10T06:01:11.234 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3136 ms) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1732 ms) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2013 ms) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1987 ms) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8868 ms total) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4033 ms) 2024-02-10T06:01:11.235 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4032 ms) 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8065 ms total) 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (71474 ms total) 2024-02-10T06:01:11.236 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-02-10T06:01:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:01:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:11.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:11.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:11.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:11.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi121-81521-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:11.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T06:01:11.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]: dispatch 2024-02-10T06:01:11.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[41707]: pgmap v155: 452 pgs: 21 creating+peering, 27 unknown, 404 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[41707]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[30020]: pgmap v155: 452 pgs: 21 creating+peering, 27 unknown, 404 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:12.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:12 smithi121 ceph-mon[30020]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:01:12.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:12 smithi146 ceph-mon[33795]: pgmap v155: 452 pgs: 21 creating+peering, 27 unknown, 404 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:12.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3132430347' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi121-81884-15"}]': finished 2024-02-10T06:01:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1039940691' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi121-81537-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:12 smithi146 ceph-mon[33795]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/923473016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/923473016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: osdmap e140: 8 total, 8 up, 8 in 2024-02-10T06:01:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/923473016' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T06:01:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: pgmap v158: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/999446249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2"}]: dispatch 2024-02-10T06:01:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: pgmap v158: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/999446249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2"}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: pgmap v158: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 op/s 2024-02-10T06:01:14.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi121-81521-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: osdmap e141: 8 total, 8 up, 8 in 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/999446249' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2"}]: dispatch 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T06:01:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi121-81682-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: osdmap e142: 8 total, 8 up, 8 in 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3844681991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3844681991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:16.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:16.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: pgmap v161: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: osdmap e143: 8 total, 8 up, 8 in 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3844681991' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:01:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:01:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:17.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81521-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: osdmap e144: 8 total, 8 up, 8 in 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3532483128' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]: dispatch 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: seed 81682 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-02-10T06:01:18.283 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (300 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (34 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (9 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (10 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-02-10T06:01:18.284 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (8 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (4 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1635 ms) 2024-02-10T06:01:18.285 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (505 ms) 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: done waiting, doing copies 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: done waiting 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61038 ms) 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (4 ms) 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4538 ms) 2024-02-10T06:01:18.286 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-10T06:01:18.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-10T06:01:18.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-10T06:01:18.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-10T06:01:18.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-10T06:01:18.287 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: pgmap v164: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[41707]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: pgmap v164: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:01:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:18 smithi121 ceph-mon[30020]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T06:01:18.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: pgmap v164: 452 pgs: 64 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:18.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi121-81682-1","app":"app1","key":"key1"}]': finished 2024-02-10T06:01:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3464390556' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi121-81537-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2483329758' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81906-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:18 smithi146 ceph-mon[33795]: osdmap e145: 8 total, 8 up, 8 in 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/874249879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/874249879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: osdmap e146: 8 total, 8 up, 8 in 2024-02-10T06:01:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/874249879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:19.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:19.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: pgmap v167: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:20.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4252216760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: pgmap v167: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:20.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T06:01:20.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4252216760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:20.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:20.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: pgmap v167: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:20.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi121-81521-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81682-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: osdmap e147: 8 total, 8 up, 8 in 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4252216760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:21.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:21.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi121-81537-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: osdmap e148: 8 total, 8 up, 8 in 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: pgmap v170: 356 pgs: 9 unknown, 347 active+clean; 592 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: pgmap v170: 356 pgs: 9 unknown, 347 active+clean; 592 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:22.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: pgmap v170: 356 pgs: 9 unknown, 347 active+clean; 592 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81682-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: osdmap e149: 8 total, 8 up, 8 in 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T06:01:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/499844876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/499844876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/392213769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi121-81521-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: osdmap e150: 8 total, 8 up, 8 in 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/499844876' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: pgmap v173: 428 pgs: 104 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:01:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: pgmap v173: 428 pgs: 104 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: pgmap v173: 428 pgs: 104 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:01:24.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:24.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi121-81906-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi121-81537-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: osdmap e151: 8 total, 8 up, 8 in 2024-02-10T06:01:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:25.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T06:01:25.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3771584548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:25.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:25.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:25.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T06:01:25.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3771584548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81682-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: osdmap e152: 8 total, 8 up, 8 in 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3771584548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: pgmap v176: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/855941200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: pgmap v176: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/855941200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: pgmap v176: 404 pgs: 80 unknown, 324 active+clean; 592 KiB data, 561 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi121-81521-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: osdmap e153: 8 total, 8 up, 8 in 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/855941200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:27.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:01:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:27.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi121-81537-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: osdmap e154: 8 total, 8 up, 8 in 2024-02-10T06:01:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP ( api_snapshots_pp: Running main() from gmock_main.cc 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: seed 81906 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (4726 ms) 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (4318 ms) 2024-02-10T06:01:28.326 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2630 ms) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2901 ms) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4199 ms) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (18774 ms total) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (5714 ms) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (6887 ms) 2024-02-10T06:01:28.327 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6782 ms) 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4001 ms) 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2040 ms) 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1361-g290212a1/rpm/el9/BUILD/ceph-19.0.0-1361-g290212a1/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi121-81906-7 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-10T06:01:28.328 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18207 ms) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (43632 ms total) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5288 ms) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5296 ms total) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3009 ms) 2024-02-10T06:01:28.329 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: pgmap v179: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: pgmap v179: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:28.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: pgmap v179: 364 pgs: 32 creating+peering, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: osdmap e155: 8 total, 8 up, 8 in 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1445071691' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]: dispatch 2024-02-10T06:01:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1825833172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1825833172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:29.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81682-24"}]': finished 2024-02-10T06:01:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/632071957' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi121-81521-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T06:01:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1825833172' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: pgmap v182: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3996535486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.032 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: pgmap v182: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:01:30.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3996535486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: pgmap v182: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi121-81537-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3996535486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:30.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1200559949' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:31.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:31.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:01:31.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1200559949' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:31.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:31.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi121-81682-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T06:01:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1200559949' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: pgmap v185: 396 pgs: 17 creating+peering, 28 unknown, 351 active+clean; 596 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/691426485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: pgmap v185: 396 pgs: 17 creating+peering, 28 unknown, 351 active+clean; 596 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/691426485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: pgmap v185: 396 pgs: 17 creating+peering, 28 unknown, 351 active+clean; 596 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:01:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi121-81521-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T06:01:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/691426485' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.327 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[41707]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1842543337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[30020]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1842543337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi121-81537-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:33 smithi146 ceph-mon[33795]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T06:01:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1842543337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/82504921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/82504921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: pgmap v188: 396 pgs: 64 unknown, 332 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:01:34.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi121-81682-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/82504921' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.336 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/264941930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/264941930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.337 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi121-81521-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/225060332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]: dispatch 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/264941930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:36.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.324 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi121-81906-16"}]': finished 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi121-81537-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:36.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:37.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:37.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2761678060' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi121-81682-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi121-81521-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]: dispatch 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:37.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: pgmap v194: 356 pgs: 32 creating+peering, 324 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 19 op/s 2024-02-10T06:01:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: pgmap v194: 356 pgs: 32 creating+peering, 324 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 19 op/s 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:38.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: pgmap v194: 356 pgs: 32 creating+peering, 324 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 19 op/s 2024-02-10T06:01:38.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-13"}]': finished 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T06:01:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:39.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:39.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi121-81521-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:39.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3848054356' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi121-81537-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T06:01:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[41707]: pgmap v197: 340 pgs: 48 unknown, 292 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[41707]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T06:01:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[30020]: pgmap v197: 340 pgs: 48 unknown, 292 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[30020]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T06:01:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:40.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:40 smithi146 ceph-mon[33795]: pgmap v197: 340 pgs: 48 unknown, 292 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:40 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:40 smithi146 ceph-mon[33795]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T06:01:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T06:01:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:41.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi121-81682-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:42.240 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosS api_aio: Running main() from gmock_main.cc 2024-02-10T06:01:42.240 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-02-10T06:01:42.240 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] Global test environment set-up. 2024-02-10T06:01:42.240 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-02-10T06:01:42.240 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10435 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6081 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (5420 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2930 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (4187 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4367 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3429 ms) 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-10T06:01:42.241 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3699 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3256 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (4035 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2734 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3020 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2996 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3228 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.Flush (2774 ms) 2024-02-10T06:01:42.242 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2993 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3034 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2969 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3025 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3022 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3258 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2786 ms) 2024-02-10T06:01:42.243 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3002 ms) 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3008 ms) 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3016 ms) 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3015 ms) 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] 26 tests from LibRadosAio (95720 ms total) 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-10T06:01:42.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: pgmap v200: 372 pgs: 23 creating+peering, 33 unknown, 316 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T06:01:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: pgmap v200: 372 pgs: 23 creating+peering, 33 unknown, 316 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:42.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: pgmap v200: 372 pgs: 23 creating+peering, 33 unknown, 316 active+clean; 585 KiB data, 600 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1026271036' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi121-81537-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3272236706' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]: dispatch 2024-02-10T06:01:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T06:01:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:43.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi121-81521-27"}]': finished 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:43.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:43.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: pgmap v203: 332 pgs: 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: pgmap v203: 332 pgs: 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]: dispatch 2024-02-10T06:01:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:44.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: pgmap v203: 332 pgs: 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-10T06:01:44.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi121-81521-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]: dispatch 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]: dispatch 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout:68111 ms total) 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (104 ms) 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (104 ms total) 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-10T06:01:44.512 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (33 ms) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (8 ms) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (42 ms total) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (51 ms) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (58 ms total) 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-10T06:01:44.513 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (39 ms) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (2 ms) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (41 ms total) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1026 ms) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1026 ms total) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104798 ms total) 2024-02-10T06:01:44.514 INFO:tasks.workunit.client.0.smithi121.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]': finished 2024-02-10T06:01:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]': finished 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T06:01:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:45.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3037383898' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi121-81682-36"}]': finished 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-15", "mode": "writeback"}]': finished 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1995479842' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi121-81537-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T06:01:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:46.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: pgmap v206: 364 pgs: 32 unknown, 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 0 op/s 2024-02-10T06:01:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:46.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T06:01:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:46.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: pgmap v206: 364 pgs: 32 unknown, 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 0 op/s 2024-02-10T06:01:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: pgmap v206: 364 pgs: 32 unknown, 332 active+clean; 601 KiB data, 600 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 4.0 KiB/s wr, 0 op/s 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi121-81521-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T06:01:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]: dispatch 2024-02-10T06:01:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:48.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: pgmap v209: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-15"}]': finished 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/437474253' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi121-81537-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:49.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T06:01:49.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3029971730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:49.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3029971730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2128500670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]: dispatch 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi121-81521-28"}]': finished 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T06:01:49.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3029971730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:49.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: pgmap v212: 300 pgs: 300 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: pgmap v212: 300 pgs: 300 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:50.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:50.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: pgmap v212: 300 pgs: 300 active+clean; 609 KiB data, 601 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 4 op/s 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi121-81537-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi121-81521-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (10434 ms) 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21960 ms) 2024-02-10T06:01:51.477 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6654 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2735 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3927 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3124 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2740 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3001 ms) 2024-02-10T06:01:51.478 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3010 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2993 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3011 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3013 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2976 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3031 ms) 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-10T06:01:51.479 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3002 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3016 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3020 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3015 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3015 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3014 ms) 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-10T06:01:51.480 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3013 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3020 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3105 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2767 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3023 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3033 ms) 2024-02-10T06:01:51.481 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]: dispatch 2024-02-10T06:01:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-10T06:01:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pg_num","val":"11"}]': finished 2024-02-10T06:01:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: pgmap v215: 364 pgs: 26 unknown, 338 active+clean; 610 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: pgmap v215: 364 pgs: 26 unknown, 338 active+clean; 610 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: pgmap v215: 364 pgs: 26 unknown, 338 active+clean; 610 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-10T06:01:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi121-81521-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:52.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi121-81602-1","var":"pgp_num","val":"11"}]': finished 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1570951947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]: dispatch 2024-02-10T06:01:52.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]: dispatch 2024-02-10T06:01:52.925 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAinapshotsECPP.SnapRemovePP (2009 ms) 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2007 ms) 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2007 ms) 2024-02-10T06:01:53.244 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9032 ms total) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (3881 ms) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4015 ms) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3960 ms) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11856 ms total) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (113475 ms total) 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-02-10T06:01:53.245 INFO:tasks.workunit.client.0.smithi121.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-02-10T06:01:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:53.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:53.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T06:01:53.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:53.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:53.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:01:53.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-17", "mode": "writeback"}]': finished 2024-02-10T06:01:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi121-81906-21"}]': finished 2024-02-10T06:01:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2350569634' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi121-81537-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T06:01:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: pgmap v218: 364 pgs: 40 unknown, 324 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 6 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: pgmap v218: 364 pgs: 40 unknown, 324 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 6 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:54.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: pgmap v218: 364 pgs: 40 unknown, 324 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 6 op/s; 31 B/s, 0 objects/s recovering 2024-02-10T06:01:54.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]: dispatch 2024-02-10T06:01:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:01:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:55.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2479059459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2479059459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:55.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-17"}]': finished 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi121-81602-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3558343986' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2479059459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]: dispatch 2024-02-10T06:01:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:56.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: pgmap v221: 323 pgs: 323 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[41707]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T06:01:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: pgmap v221: 323 pgs: 323 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:56 smithi121 ceph-mon[30020]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T06:01:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: pgmap v221: 323 pgs: 323 active+clean; 585 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:01:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:01:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi121-81521-29"}]': finished 2024-02-10T06:01:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi121-81537-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3800042326' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-10T06:01:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:56 smithi146 ceph-mon[33795]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T06:01:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi121-81521-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:57.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: pgmap v224: 323 pgs: 32 creating+peering, 1 peering, 290 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006915477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: pgmap v224: 323 pgs: 32 creating+peering, 1 peering, 290 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006915477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: pgmap v224: 323 pgs: 32 creating+peering, 1 peering, 290 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:01:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi121-81602-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006915477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:01:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:01:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T06:01:59.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:01:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:01:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:01:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi121-81521-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:01:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:01:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi121-81537-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:01:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T06:01:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:01:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: pgmap v227: 324 pgs: 64 unknown, 260 active+clean; 583 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T06:02:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]: dispatch 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: pgmap v227: 324 pgs: 64 unknown, 260 active+clean; 583 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T06:02:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]: dispatch 2024-02-10T06:02:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: pgmap v227: 324 pgs: 64 unknown, 260 active+clean; 583 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:02:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi121-81602-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T06:02:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]: dispatch 2024-02-10T06:02:01.317 INFO:tasks.workunit.client.0.smithi121.stdout:expected=12 2024-02-10T06:02:01.317 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:d83876eb:::4:head 2024-02-10T06:02:01.317 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:d83876eb:::4:head expected=11:d83876eb:::4:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:d83876eb:::4:head -> 4 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=4 expected=4 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:b29083e3:::5:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:b29083e3:::5:head expected=11:b29083e3:::5:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:b29083e3:::5:head -> 5 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=5 expected=5 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:89d3ae78:::11:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:89d3ae78:::11:head expected=11:89d3ae78:::11:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=11 expected=11 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:863748b0:::15:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:863748b0:::15:head expected=11:863748b0:::15:head 2024-02-10T06:02:01.318 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:863748b0:::15:head -> 15 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=15 expected=15 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:6cac518f:::0:head expected=11:6cac518f:::0:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=0 expected=0 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:62a1935d:::14:head expected=11:62a1935d:::14:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=14 expected=14 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:52ea6a34:::10:head expected=11:52ea6a34:::10:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=10 expected=10 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-02-10T06:02:01.319 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : cursor()=11:02547ec2:::1:head expected=11:02547ec2:::1:head 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: : entry=1 expected=1 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (198 ms) 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (100430 ms) 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: split 0/5 -> MIN 11:33333333::::head 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: split 1/5 -> 11:33333333::::head 11:66666666::::head 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: split 2/5 -> 11:66666666::::head 11:99999999::::head 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: split 3/5 -> 11:99999999::::head 11:cccccccc::::head 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: split 4/5 -> 11:cccccccc::::head MAX 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (6016 ms) 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 7 tests from LibRadosList (108173 ms total) 2024-02-10T06:02:01.320 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1030 ms) 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo1,foo2,foo3 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo1 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo2 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo3 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo1,foo4,foo5 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo4 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo5 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo1 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset foo6,foo7 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo7 2024-02-10T06:02:01.321 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: foo6 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo4 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo5 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns2:foo7 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns2:foo6 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: ns1:foo1 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo1 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo2 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: :foo3 2024-02-10T06:02:01.322 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (30 ms) 2024-02-10T06:02:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: pgmap v230: 308 pgs: 3 creating+peering, 19 unknown, 286 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]': finished 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: pgmap v230: 308 pgs: 3 creating+peering, 19 unknown, 286 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]': finished 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: pgmap v230: 308 pgs: 3 creating+peering, 19 unknown, 286 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-19", "mode": "writeback"}]': finished 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:02.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T06:02:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "prefix": "osd pool set", 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "var": "eio", 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "val": "true" 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: }]: dispatch 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "prefix": "osd pool set", 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "var": "eio", 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: "val": "true" 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[41707]: }]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "prefix": "osd pool set", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "var": "eio", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "val": "true" 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: }]: dispatch 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "prefix": "osd pool set", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "var": "eio", 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: "val": "true" 2024-02-10T06:02:03.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:03 smithi121 ceph-mon[30020]: }]: dispatch 2024-02-10T06:02:03.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi121-81537-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2828878957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]: dispatch 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]: dispatch 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3647740540' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "prefix": "osd pool set", 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "var": "eio", 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "val": "true" 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: }]: dispatch 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{ 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "prefix": "osd pool set", 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "var": "eio", 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: "val": "true" 2024-02-10T06:02:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:03 smithi146 ceph-mon[33795]: }]: dispatch 2024-02-10T06:02:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: pgmap v233: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{ 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: "prefix": "osd pool set", 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: "var": "eio", 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: "val": "true" 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: }]': finished 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: pgmap v233: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{ 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: "prefix": "osd pool set", 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:04.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: "var": "eio", 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: "val": "true" 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: }]': finished 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:04.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: pgmap v233: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-19"}]': finished 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi121-81521-30"}]': finished 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{ 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: "prefix": "osd pool set", 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: "pool": "PoolEIOFlag_smithi121-81537-33", 2024-02-10T06:02:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: "var": "eio", 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: "val": "true" 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: }]': finished 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/121088633' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:04.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[41707]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:05 smithi121 ceph-mon[30020]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T06:02:05.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi121-81602-2"}]': finished 2024-02-10T06:02:05.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi121-81521-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T06:02:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:05 smithi146 ceph-mon[33795]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T06:02:06.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: pgmap v236: 260 pgs: 260 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/622772646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: pgmap v236: 260 pgs: 260 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/622772646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:06.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: pgmap v236: 260 pgs: 260 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/622772646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi121-81521-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81602-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi121-81537-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T06:02:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/429040067' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:06.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 1 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 10 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 13 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 7 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 14 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 0 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 15 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 11 0 2024-02-10T06:02:07.305 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 5 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 8 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 6 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 3 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 4 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 12 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 9 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 2 0 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (42 ms) 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 3 tests from LibRadosListEC (1102 ms total) 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2765 ms) 2024-02-10T06:02:07.306 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] 1 test from LibRadosListNP (2765 ms total) 2024-02-10T06:02:07.307 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: 2024-02-10T06:02:07.307 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [----------] Global test environment tear-down 2024-02-10T06:02:07.307 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [==========] 11 tests from 3 test suites ran. (127614 ms total) 2024-02-10T06:02:07.307 INFO:tasks.workunit.client.0.smithi121.stdout: api_list: [ PASSED ] 11 tests. 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[41707]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T06:02:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[30020]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T06:02:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi121-81602-3","pool2":"test-rados-api-smithi121-81602-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-10T06:02:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:07 smithi146 ceph-mon[33795]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T06:02:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: pgmap v239: 364 pgs: 8 unknown, 96 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1300946120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]: dispatch 2024-02-10T06:02:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: pgmap v239: 364 pgs: 8 unknown, 96 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1300946120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: pgmap v239: 364 pgs: 8 unknown, 96 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1300946120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]: dispatch 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: pgmap v242: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]': finished 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: pgmap v242: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]': finished 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:10.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: pgmap v242: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-21", "mode": "writeback"}]': finished 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi121-81537-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2452490553' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]: dispatch 2024-02-10T06:02:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T06:02:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:11.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:11.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi121-81521-31"}]': finished 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: pgmap v245: 292 pgs: 13 creating+peering, 279 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1241527028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: pgmap v245: 292 pgs: 13 creating+peering, 279 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1241527028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: pgmap v245: 292 pgs: 13 creating+peering, 279 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-21"}]': finished 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi121-81521-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1241527028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1696047446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1696047446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T06:02:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi121-81521-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T06:02:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1696047446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:14 smithi146 ceph-mon[33795]: pgmap v248: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:14 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:14 smithi146 ceph-mon[33795]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T06:02:15.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[41707]: pgmap v248: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:15.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[41707]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[30020]: pgmap v248: 292 pgs: 32 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:14 smithi121 ceph-mon[30020]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T06:02:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:16.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: pgmap v251: 364 pgs: 104 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: pgmap v251: 364 pgs: 104 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: pgmap v251: 364 pgs: 104 unknown, 260 active+clean; 583 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:17.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:17.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/323920366' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi121-81537-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:17.731 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T06:02:17.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:17.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3043113375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]: dispatch 2024-02-10T06:02:17.732 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:18.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]': finished 2024-02-10T06:02:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:18.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T06:02:18.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T06:02:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]': finished 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi121-81521-32"}]': finished 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-23", "mode": "writeback"}]': finished 2024-02-10T06:02:18.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi121-81521-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:18.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T06:02:18.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:18.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:20 smithi146 ceph-mon[33795]: pgmap v257: 324 pgs: 324 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:20 smithi146 ceph-mon[33795]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T06:02:20.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[41707]: pgmap v257: 324 pgs: 324 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[41707]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[30020]: pgmap v257: 324 pgs: 324 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[30020]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T06:02:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:21 smithi146 ceph-mon[33795]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T06:02:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[41707]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi121-81521-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[30020]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T06:02:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]: dispatch 2024-02-10T06:02:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:22.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: pgmap v260: 332 pgs: 24 creating+peering, 16 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:02:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:22 smithi146 ceph-mon[33795]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: pgmap v260: 332 pgs: 24 creating+peering, 16 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[30020]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: pgmap v260: 332 pgs: 24 creating+peering, 16 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-23"}]': finished 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1192922953' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:22 smithi121 ceph-mon[41707]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T06:02:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T06:02:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2764569190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:23 smithi146 ceph-mon[33795]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2764569190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[30020]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2764569190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:23 smithi121 ceph-mon[41707]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: pgmap v263: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:24.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: pgmap v263: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: pgmap v263: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/183776996' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi121-81521-33"}]': finished 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:24.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:26.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: pgmap v266: 388 pgs: 96 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: pgmap v266: 388 pgs: 96 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: pgmap v266: 388 pgs: 96 unknown, 32 creating+peering, 260 active+clean; 583 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4255360744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi121-81521-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T06:02:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:27.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2763948227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2763948227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:27.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T06:02:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2763948227' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: pgmap v269: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]: dispatch 2024-02-10T06:02:28.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: pgmap v269: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T06:02:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]: dispatch 2024-02-10T06:02:28.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: pgmap v269: 420 pgs: 32 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:28.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi121-81521-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:28.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:28.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:28.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T06:02:28.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]: dispatch 2024-02-10T06:02:29.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:29.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]': finished 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2863757456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:29.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]': finished 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2863757456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:29.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T06:02:29.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:29.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:29.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:29.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-25", "mode": "writeback"}]': finished 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2863757456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi121-81537-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:29.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]: dispatch 2024-02-10T06:02:29.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: pgmap v272: 460 pgs: 72 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: pgmap v272: 460 pgs: 72 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: pgmap v272: 460 pgs: 72 unknown, 388 active+clean; 584 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:02:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-25"}]': finished 2024-02-10T06:02:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T06:02:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/688029833' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]: dispatch 2024-02-10T06:02:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[41707]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T06:02:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:31 smithi121 ceph-mon[30020]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T06:02:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi121-81521-34"}]': finished 2024-02-10T06:02:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:31 smithi146 ceph-mon[33795]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T06:02:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: pgmap v275: 420 pgs: 15 unknown, 405 active+clean; 584 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: pgmap v275: 420 pgs: 15 unknown, 405 active+clean; 584 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: pgmap v275: 420 pgs: 15 unknown, 405 active+clean; 584 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi121-81521-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[41707]: pgmap v278: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[41707]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[30020]: pgmap v278: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:34.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:34 smithi121 ceph-mon[30020]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T06:02:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:34 smithi146 ceph-mon[33795]: pgmap v278: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:02:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:34 smithi146 ceph-mon[33795]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T06:02:35.277 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[41707]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[30020]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T06:02:35.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:35.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi121-81521-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:35.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:35 smithi146 ceph-mon[33795]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T06:02:35.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:36.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-mon[33795]: pgmap v281: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:36.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:36.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-mon[33795]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T06:02:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:36.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[41707]: pgmap v281: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[41707]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T06:02:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[30020]: pgmap v281: 300 pgs: 40 unknown, 260 active+clean; 583 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[30020]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T06:02:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:36.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]': finished 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]: dispatch 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]': finished 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/733007885' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi121-81537-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]: dispatch 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-27", "mode": "writeback"}]': finished 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T06:02:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2872695914' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:37.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]: dispatch 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: pgmap v284: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3884076940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:38.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: pgmap v284: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:02:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T06:02:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3884076940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:38.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: pgmap v284: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:02:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi121-81521-35"}]': finished 2024-02-10T06:02:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T06:02:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3884076940' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T06:02:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:39.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi121-81537-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi121-81521-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]: dispatch 2024-02-10T06:02:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[41707]: pgmap v287: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:02:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[30020]: pgmap v287: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:02:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:40 smithi146 ceph-mon[33795]: pgmap v287: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:02:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:40 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-27"}]': finished 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T06:02:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi121-81521-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T06:02:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[41707]: pgmap v290: 292 pgs: 292 active+clean; 584 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[41707]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T06:02:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[30020]: pgmap v290: 292 pgs: 292 active+clean; 584 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[30020]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T06:02:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:42.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:42 smithi146 ceph-mon[33795]: pgmap v290: 292 pgs: 292 active+clean; 584 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:42 smithi146 ceph-mon[33795]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T06:02:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T06:02:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:44.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: pgmap v293: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:02:44.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T06:02:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]: dispatch 2024-02-10T06:02:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:46.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: pgmap v296: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: pgmap v296: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T06:02:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: pgmap v296: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3134362841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi121-81521-36"}]': finished 2024-02-10T06:02:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:46.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T06:02:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:47.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:47.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]: dispatch 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2115047858' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-46"}]': finished 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: pgmap v299: 292 pgs: 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]: dispatch 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]': finished 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: pgmap v299: 292 pgs: 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]: dispatch 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]': finished 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T06:02:48.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: pgmap v299: 292 pgs: 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T06:02:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]: dispatch 2024-02-10T06:02:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi121-81521-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-29", "mode": "writeback"}]': finished 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi121-81537-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T06:02:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:49.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:49 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:49.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:49 smithi121 ceph-mon[30020]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T06:02:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:49 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:49 smithi121 ceph-mon[41707]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T06:02:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:49 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:49 smithi146 ceph-mon[33795]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T06:02:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[30020]: pgmap v302: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[30020]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T06:02:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[41707]: pgmap v302: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[41707]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T06:02:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:50 smithi146 ceph-mon[33795]: pgmap v302: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:02:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi121-81537-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:50 smithi146 ceph-mon[33795]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T06:02:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: pgmap v305: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: pgmap v305: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:52.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: pgmap v305: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T06:02:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1248655795' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]: dispatch 2024-02-10T06:02:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:53.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi121-81521-37"}]': finished 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:53.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: pgmap v308: 292 pgs: 292 active+clean; 8.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: pgmap v308: 292 pgs: 292 active+clean; 8.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:55.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T06:02:55.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:55.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: pgmap v308: 292 pgs: 292 active+clean; 8.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-29"}]': finished 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi121-81521-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]: dispatch 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2364688131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi121-81537-47"}]': finished 2024-02-10T06:02:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:02:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: pgmap v311: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: pgmap v311: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:56.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T06:02:56.150 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: pgmap v311: 260 pgs: 260 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi121-81521-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi121-81537-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:56.151 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:02:56.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:02:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:57 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:57 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: pgmap v314: 300 pgs: 40 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: pgmap v314: 300 pgs: 40 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:58.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:58 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:58.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: pgmap v314: 300 pgs: 40 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:02:58.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi121-81537-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:02:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:58 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]: dispatch 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]': finished 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]: dispatch 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:59.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]': finished 2024-02-10T06:02:59.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T06:02:59.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:59.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:02:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1006104112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]: dispatch 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]: dispatch 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-31", "mode": "writeback"}]': finished 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi121-81521-38"}]': finished 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:02:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:02:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: pgmap v317: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: pgmap v317: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:03:00.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:03:00.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: pgmap v317: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi121-81521-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3863462511' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]: dispatch 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]: dispatch 2024-02-10T06:03:00.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: pgmap v320: 292 pgs: 7 creating+peering, 285 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:03:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: pgmap v320: 292 pgs: 7 creating+peering, 285 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:02.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:02.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: pgmap v320: 292 pgs: 7 creating+peering, 285 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-31"}]': finished 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi121-81537-48"}]': finished 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[41707]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T06:03:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:03.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:03.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:03.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:03.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[30020]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T06:03:03.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:03.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:03.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi121-81521-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi121-81537-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:03 smithi146 ceph-mon[33795]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T06:03:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[41707]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[41707]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T06:03:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:04.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[30020]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:04.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:04.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[30020]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T06:03:04.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:04 smithi146 ceph-mon[33795]: pgmap v323: 268 pgs: 8 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:04 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:04 smithi146 ceph-mon[33795]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T06:03:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:05.284 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:05.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi121-81537-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T06:03:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:06.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: pgmap v326: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:06.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: pgmap v326: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: pgmap v326: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/616215088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]: dispatch 2024-02-10T06:03:07.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:07.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:07.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]': finished 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:07.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]': finished 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi121-81521-39"}]': finished 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]: dispatch 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-33", "mode": "writeback"}]': finished 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi121-81521-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2759358663' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:07.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]: dispatch 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[41707]: pgmap v329: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[30020]: pgmap v329: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:08 smithi146 ceph-mon[33795]: pgmap v329: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi121-81537-49"}]': finished 2024-02-10T06:03:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T06:03:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:09.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi121-81521-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-33"}]': finished 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi121-81537-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:10 smithi121 ceph-mon[41707]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:10 smithi121 ceph-mon[41707]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T06:03:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:10 smithi121 ceph-mon[30020]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:10 smithi121 ceph-mon[30020]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T06:03:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:10 smithi146 ceph-mon[33795]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:10 smithi146 ceph-mon[33795]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T06:03:12.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[41707]: pgmap v335: 268 pgs: 1 unknown, 267 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:03:12.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:12.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[41707]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T06:03:12.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:12.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:12.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[30020]: pgmap v335: 268 pgs: 1 unknown, 267 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:03:12.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:12.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[30020]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T06:03:12.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:12.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:12.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:12 smithi146 ceph-mon[33795]: pgmap v335: 268 pgs: 1 unknown, 267 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:03:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi121-81537-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:12 smithi146 ceph-mon[33795]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T06:03:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:12.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T06:03:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:13.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:13.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:13.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T06:03:13.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:13.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:13.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T06:03:13.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]: dispatch 2024-02-10T06:03:13.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:13.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3734169743' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi121-81521-40"}]': finished 2024-02-10T06:03:13.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:13.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T06:03:13.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:14.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: pgmap v338: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:14.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]: dispatch 2024-02-10T06:03:14.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: pgmap v338: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: pgmap v338: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi121-81521-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]: dispatch 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3665735185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]: dispatch 2024-02-10T06:03:14.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:15 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]': finished 2024-02-10T06:03:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:15 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]': finished 2024-02-10T06:03:16.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:15 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:16.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-35", "mode": "writeback"}]': finished 2024-02-10T06:03:16.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: pgmap v341: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2869946844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: pgmap v341: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T06:03:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2869946844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:17.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: pgmap v341: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi121-81537-50"}]': finished 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi121-81521-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T06:03:17.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2869946844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:17.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:18.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[41707]: pgmap v344: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[41707]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[30020]: pgmap v344: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[30020]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T06:03:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:18.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:18 smithi146 ceph-mon[33795]: pgmap v344: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi121-81537-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:18 smithi146 ceph-mon[33795]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T06:03:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:19.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:19.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:19 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:19.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:19.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:19 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3836652980' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]: dispatch 2024-02-10T06:03:21.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:21.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:21.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T06:03:21.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:21.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:21.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi121-81521-41"}]': finished 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi121-81537-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi121-81521-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]: dispatch 2024-02-10T06:03:22.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:22 smithi146 ceph-mon[33795]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:22 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:22 smithi146 ceph-mon[33795]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T06:03:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[41707]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[41707]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[30020]: pgmap v350: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-35"}]': finished 2024-02-10T06:03:23.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:22 smithi121 ceph-mon[30020]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T06:03:24.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: pgmap v353: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi121-81521-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]: dispatch 2024-02-10T06:03:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: pgmap v356: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:03:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: pgmap v356: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:03:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:26.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:26.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: pgmap v356: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1036639857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi121-81537-52"}]': finished 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:26.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:26.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:27.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6879 ms) 2024-02-10T06:03:27.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-02-10T06:03:27.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6991 ms) 2024-02-10T06:03:27.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-10T06:03:27.270 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7027 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7015 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7033 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7030 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6915 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7019 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7024 ms) 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-10T06:03:27.271 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6956 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7024 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6987 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7047 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7001 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7023 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6934 ms) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (111906 ms total) 2024-02-10T06:03:27.272 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: 2024-02-10T06:03:27.273 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [----------] Global test environment tear-down 2024-02-10T06:03:27.273 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (207627 ms total) 2024-02-10T06:03:27.273 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio: [ PASSED ] 42 tests. 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T06:03:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T06:03:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:27.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2070305375' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]: dispatch 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]': finished 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]: dispatch 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]': finished 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T06:03:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:28.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: pgmap v359: 292 pgs: 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi121-81521-42"}]': finished 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]: dispatch 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi121-81537-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-37", "mode": "writeback"}]': finished 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T06:03:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[41707]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T06:03:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:29.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:29.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:29.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[30020]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T06:03:29.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:29 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:29 smithi146 ceph-mon[33795]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T06:03:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]: dispatch 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: pgmap v362: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: pgmap v362: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T06:03:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: pgmap v362: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-37"}]': finished 2024-02-10T06:03:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T06:03:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: pgmap v365: 292 pgs: 292 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4181387705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]: dispatch 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:33.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi121-81537-53"}]': finished 2024-02-10T06:03:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T06:03:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: pgmap v368: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: pgmap v368: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:34 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:34.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: pgmap v368: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:03:34.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi121-81537-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T06:03:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:34 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T06:03:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]: dispatch 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:35.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T06:03:35.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]: dispatch 2024-02-10T06:03:35.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:35.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T06:03:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi121-81537-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T06:03:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]: dispatch 2024-02-10T06:03:36.533 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[41707]: pgmap v371: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]': finished 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[41707]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[30020]: pgmap v371: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]': finished 2024-02-10T06:03:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[30020]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T06:03:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:36.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-mon[33795]: pgmap v371: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:36.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:03:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-39", "mode": "writeback"}]': finished 2024-02-10T06:03:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-mon[33795]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T06:03:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[41707]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[30020]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:03:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:37 smithi146 ceph-mon[33795]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T06:03:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]: dispatch 2024-02-10T06:03:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: pgmap v374: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: pgmap v374: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T06:03:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: pgmap v374: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:03:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-39"}]': finished 2024-02-10T06:03:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T06:03:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]: dispatch 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: seed 81810 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (23 ms) 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-10T06:03:39.295 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (13642 ms) 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (2 ms) 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (13667 ms total) 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7735 ms) 2024-02-10T06:03:39.296 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8142 ms) 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9796 ms) 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: my_snaps [3] 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: my_snaps [4,3] 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: my_snaps [5,4,3] 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: promoting some heads 2024-02-10T06:03:39.297 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: promoting from clones for snap 3 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: waiting for scrubs... 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: done waiting 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46147 ms) 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9924 ms) 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7969 ms) 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-10T06:03:39.298 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8026 ms) 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8279 ms) 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9780 ms) 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8918 ms) 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10032 ms) 2024-02-10T06:03:39.299 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13192 ms) 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7779 ms) 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8268 ms) 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12803 ms) 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7929 ms) 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-10T06:03:39.300 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8025 ms) 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:40 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3456717449' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi121-81537-54"}]': finished 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:40 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:41.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi121-81537-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T06:03:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:41.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: pgmap v380: 292 pgs: 30 creating+peering, 2 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T06:03:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: pgmap v380: 292 pgs: 30 creating+peering, 2 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T06:03:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:03:42.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: pgmap v380: 292 pgs: 30 creating+peering, 2 unknown, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:03:42.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T06:03:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi121-81537-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T06:03:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:03:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[41707]: pgmap v383: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:03:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:03:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[41707]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T06:03:44.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[30020]: pgmap v383: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:03:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:03:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[30020]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T06:03:44.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:44.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:44 smithi146 ceph-mon[33795]: pgmap v383: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:03:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:03:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:44 smithi146 ceph-mon[33795]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T06:03:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:45.317 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:45.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:45.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T06:03:45.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:03:45.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[41707]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:45.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:45 smithi121 ceph-mon[30020]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T06:03:45.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:45.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:45 smithi146 ceph-mon[33795]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: pgmap v386: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[41707]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: pgmap v386: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:46 smithi121 ceph-mon[30020]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T06:03:46.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: pgmap v386: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/109352935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]: dispatch 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi121-81537-55"}]': finished 2024-02-10T06:03:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:46 smithi146 ceph-mon[33795]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]: dispatch 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: pgmap v389: 292 pgs: 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]': finished 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: pgmap v389: 292 pgs: 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]': finished 2024-02-10T06:03:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T06:03:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: pgmap v389: 292 pgs: 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi121-81537-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-41"}]': finished 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:49.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:49 smithi146 ceph-mon[33795]: osdmap e296: 8 total, 8 up, 8 in 2024-02-10T06:03:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:49 smithi121 ceph-mon[41707]: osdmap e296: 8 total, 8 up, 8 in 2024-02-10T06:03:49.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:49 smithi121 ceph-mon[30020]: osdmap e296: 8 total, 8 up, 8 in 2024-02-10T06:03:50.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:50 smithi146 ceph-mon[33795]: pgmap v392: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:50 smithi146 ceph-mon[33795]: osdmap e297: 8 total, 8 up, 8 in 2024-02-10T06:03:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[41707]: pgmap v392: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[41707]: osdmap e297: 8 total, 8 up, 8 in 2024-02-10T06:03:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[30020]: pgmap v392: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi121-81537-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[30020]: osdmap e297: 8 total, 8 up, 8 in 2024-02-10T06:03:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:51.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: osdmap e298: 8 total, 8 up, 8 in 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: osdmap e299: 8 total, 8 up, 8 in 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:51.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: osdmap e298: 8 total, 8 up, 8 in 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: osdmap e299: 8 total, 8 up, 8 in 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T06:03:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: osdmap e298: 8 total, 8 up, 8 in 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-6","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: osdmap e299: 8 total, 8 up, 8 in 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: pgmap v395: 300 pgs: 3 unknown, 297 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]': finished 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: osdmap e300: 8 total, 8 up, 8 in 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: pgmap v395: 300 pgs: 3 unknown, 297 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]': finished 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: osdmap e300: 8 total, 8 up, 8 in 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: pgmap v395: 300 pgs: 3 unknown, 297 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_count","val": "8"}]': finished 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: osdmap e300: 8 total, 8 up, 8 in 2024-02-10T06:03:52.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/969007700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:03:52.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]: dispatch 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: ok, hit_set contains 263:602f83fe:::foo:head 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9288 ms) 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg_num = 32 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 0 ls 1707545035,0 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 1 ls 1707545035,0 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 2 ls 1707545035,0 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 3 ls 1707545035,0 2024-02-10T06:03:54.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 4 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 5 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 6 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 7 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 8 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 9 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 10 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 11 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 12 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 13 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 14 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 15 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 16 ls 1707545035,0 2024-02-10T06:03:54.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 17 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 18 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 19 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 20 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 21 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 22 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 23 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 24 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 25 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 26 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 27 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 28 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 29 ls 1707545035,0 2024-02-10T06:03:54.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 30 ls 1707545035,0 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg 31 ls 1707545035,0 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: pg_num = 32 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6cac518f:::0:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:02547ec2:::1:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f905c69b:::2:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:cfc208b3:::3:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d83876eb:::4:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b29083e3:::5:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c4fdafeb:::6:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:5c6b0b28:::7:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:bd63b0f1:::8:head 2024-02-10T06:03:54.596 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e960b815:::9:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:52ea6a34:::10:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:89d3ae78:::11:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:de5d7c5f:::12:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:566253c9:::13:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:62a1935d:::14:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:863748b0:::15:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3958e169:::16:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4d4dabf9:::17:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:8391935d:::18:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:28883081:::19:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:69259c59:::20:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4bdb80b7:::21:head 2024-02-10T06:03:54.597 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a11c5d71:::22:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:271af37b:::23:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:95b121be:::24:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:58d1031b:::25:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:0a050783:::26:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c709704c:::27:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:cbe56eaf:::28:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:86b4b162:::29:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:70d89383:::30:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:dd450c7c:::31:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6d5729b1:::32:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c388f3fb:::33:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:56cfea31:::34:head 2024-02-10T06:03:54.598 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9dbc1bf7:::35:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:40b74ccd:::36:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4d5aaf42:::37:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:920f362c:::38:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6cc53222:::39:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9cad833f:::40:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:1ea84d41:::41:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c4480ef6:::42:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a694361e:::43:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d1bd33e9:::44:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ddc2cd5d:::45:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:2b782207:::46:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7b187fca:::47:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:90ecdf6f:::48:head 2024-02-10T06:03:54.599 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a5ed95fe:::49:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ea0eaa55:::50:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f33ef17b:::51:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a0d1b2f6:::52:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:60c5229e:::53:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:edcbc575:::54:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:102cf253:::55:head 2024-02-10T06:03:54.600 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:efb7fb0b:::56:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:50d0a326:::57:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d4dc5daf:::58:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3a130462:::59:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ec87ed71:::60:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d5bc9454:::61:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3ddfe313:::62:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7c2816b9:::63:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:47e00e4d:::64:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c6410c18:::65:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b48ed237:::66:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:cd63ad31:::67:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b179e92b:::68:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:0d9f741a:::69:head 2024-02-10T06:03:54.601 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6d3352ae:::70:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c6d5c19e:::71:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:bc4729c3:::72:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:77e930b9:::73:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:0abeecfd:::74:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b7c37e15:::75:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b6378398:::76:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:02bd68de:::77:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:cc795d2d:::78:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:630d4fea:::79:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e0d29ef5:::80:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:fd6f13d2:::81:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:606461d5:::82:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:eadbdc43:::83:head 2024-02-10T06:03:54.602 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:8761d0bb:::84:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9ef0186f:::85:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e0d41294:::86:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:961de695:::87:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:1423148f:::88:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:633a8fa2:::89:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a8653809:::90:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3dac8b33:::91:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:35aad435:::92:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f6dcc343:::93:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:dbbdad87:::94:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:1cb48ce0:::95:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:03cd461c:::96:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:17a4ea99:::97:head 2024-02-10T06:03:54.603 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9993c9a7:::98:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6394211c:::99:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:94c7ae57:::100:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6fdee5bb:::101:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9a477fd1:::102:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:eb850916:::103:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:affc56b9:::104:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b42dc814:::105:head 2024-02-10T06:03:54.604 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f319f8f0:::106:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9a40b9de:::107:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:8b524f28:::108:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e3de589f:::109:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:90f90a5b:::110:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a7b4f1d7:::111:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:af51766e:::112:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b6f90bd1:::113:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e0261208:::114:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c9569ef7:::115:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:61bebe50:::116:head 2024-02-10T06:03:54.605 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:fe93412b:::117:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d3d38bee:::118:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3100ba0c:::119:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d0560ada:::120:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f0ea8b35:::121:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:766f231a:::122:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a07a2582:::123:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:bd7c6b3a:::124:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:fb2ddaff:::125:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4408e1fe:::126:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ee1df7a7:::127:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c3002909:::128:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4f48ffa9:::129:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:edf38733:::130:head 2024-02-10T06:03:54.606 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c08425c0:::131:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:5f902d98:::132:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:41ea2c93:::133:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:813cee13:::134:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:0131818d:::135:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:26ba5a85:::136:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:381b8a5a:::137:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:28797e47:::138:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:bfca7f22:::139:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:36807075:::140:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:80b03975:::141:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:5c15709b:::142:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f39ea15e:::143:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ea992956:::144:head 2024-02-10T06:03:54.607 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:48887b1c:::145:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:9f24a9dd:::146:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:987f100b:::147:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d2dd3581:::148:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7fed1808:::149:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c80b70e9:::150:head 2024-02-10T06:03:54.608 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:85ed90f9:::151:head 2024-02-10T06:03:54.612 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:36428b24:::152:head 2024-02-10T06:03:54.612 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d044c34a:::153:head 2024-02-10T06:03:54.612 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7c18bf58:::154:head 2024-02-10T06:03:54.612 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d1c21232:::155:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a7a3c575:::156:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:87da0633:::157:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d5ac3822:::158:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3f20522d:::159:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6ca26563:::160:head 2024-02-10T06:03:54.613 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:532ce135:::161:head 2024-02-10T06:03:54.614 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:c78863e6:::162:head 2024-02-10T06:03:54.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: osdmap e301: 8 total, 8 up, 8 in 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:54 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: osdmap e301: 8 total, 8 up, 8 in 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:03:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi121-81537-56"}]': finished 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: osdmap e301: 8 total, 8 up, 8 in 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:54.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:54 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:03:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:03:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: osdmap e302: 8 total, 8 up, 8 in 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-43","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:55.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]: dispatch 2024-02-10T06:03:55.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]': finished 2024-02-10T06:03:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T06:03:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: osdmap e302: 8 total, 8 up, 8 in 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-43","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]: dispatch 2024-02-10T06:03:56.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]': finished 2024-02-10T06:03:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-10T06:03:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi121-81537-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:03:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: osdmap e302: 8 total, 8 up, 8 in 2024-02-10T06:03:56.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:56.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:56.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi121-81810-43","var": "pg_num","format": "json"}]: dispatch 2024-02-10T06:03:56.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:03:56.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]: dispatch 2024-02-10T06:03:56.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-43"}]': finished 2024-02-10T06:03:56.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:03:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:03:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:56 smithi146 ceph-mon[33795]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:56 smithi146 ceph-mon[33795]: osdmap e303: 8 total, 8 up, 8 in 2024-02-10T06:03:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:03:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:56 smithi146 ceph-mon[33795]: osdmap e304: 8 total, 8 up, 8 in 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[41707]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[41707]: osdmap e303: 8 total, 8 up, 8 in 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[41707]: osdmap e304: 8 total, 8 up, 8 in 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[30020]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:03:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[30020]: osdmap e303: 8 total, 8 up, 8 in 2024-02-10T06:03:57.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi121-81537-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:03:57.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:56 smithi121 ceph-mon[30020]: osdmap e304: 8 total, 8 up, 8 in 2024-02-10T06:03:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:57 smithi146 ceph-mon[33795]: osdmap e305: 8 total, 8 up, 8 in 2024-02-10T06:03:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[30020]: osdmap e305: 8 total, 8 up, 8 in 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[41707]: osdmap e305: 8 total, 8 up, 8 in 2024-02-10T06:03:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:03:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: pgmap v404: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: osdmap e306: 8 total, 8 up, 8 in 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:58 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: pgmap v404: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: osdmap e306: 8 total, 8 up, 8 in 2024-02-10T06:03:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: pgmap v404: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:03:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: osdmap e306: 8 total, 8 up, 8 in 2024-02-10T06:03:59.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:03:59.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:59.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:59.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:58 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: osdmap e307: 8 total, 8 up, 8 in 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:03:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:03:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: osdmap e307: 8 total, 8 up, 8 in 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:04:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: osdmap e307: 8 total, 8 up, 8 in 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2691369272' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:04:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:03:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]: dispatch 2024-02-10T06:04:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: pgmap v407: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: osdmap e308: 8 total, 8 up, 8 in 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: pgmap v407: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: osdmap e308: 8 total, 8 up, 8 in 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: pgmap v407: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi121-81537-57"}]': finished 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: osdmap e308: 8 total, 8 up, 8 in 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:01.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:04:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: osdmap e309: 8 total, 8 up, 8 in 2024-02-10T06:04:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: osdmap e309: 8 total, 8 up, 8 in 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi121-81537-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: osdmap e309: 8 total, 8 up, 8 in 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:02 smithi146 ceph-mon[33795]: pgmap v410: 292 pgs: 4 unknown, 288 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:02 smithi146 ceph-mon[33795]: osdmap e310: 8 total, 8 up, 8 in 2024-02-10T06:04:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[41707]: pgmap v410: 292 pgs: 4 unknown, 288 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[41707]: osdmap e310: 8 total, 8 up, 8 in 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[30020]: pgmap v410: 292 pgs: 4 unknown, 288 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[30020]: osdmap e310: 8 total, 8 up, 8 in 2024-02-10T06:04:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:04:04.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:04 smithi146 ceph-mon[33795]: pgmap v413: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:04:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:04 smithi146 ceph-mon[33795]: osdmap e311: 8 total, 8 up, 8 in 2024-02-10T06:04:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:04 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[41707]: pgmap v413: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:04.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:04.842 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[41707]: osdmap e311: 8 total, 8 up, 8 in 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[30020]: pgmap v413: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi121-81537-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[30020]: osdmap e311: 8 total, 8 up, 8 in 2024-02-10T06:04:04.843 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:04 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:05.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:05 smithi146 ceph-mon[33795]: osdmap e312: 8 total, 8 up, 8 in 2024-02-10T06:04:05.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:05 smithi121 ceph-mon[30020]: osdmap e312: 8 total, 8 up, 8 in 2024-02-10T06:04:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:05 smithi121 ceph-mon[41707]: osdmap e312: 8 total, 8 up, 8 in 2024-02-10T06:04:06.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:06 smithi146 ceph-mon[33795]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:06.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:06 smithi146 ceph-mon[33795]: osdmap e313: 8 total, 8 up, 8 in 2024-02-10T06:04:06.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:06.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[41707]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[41707]: osdmap e313: 8 total, 8 up, 8 in 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[30020]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[30020]: osdmap e313: 8 total, 8 up, 8 in 2024-02-10T06:04:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:07.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: osdmap e314: 8 total, 8 up, 8 in 2024-02-10T06:04:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:07 smithi146 ceph-mon[33795]: osdmap e315: 8 total, 8 up, 8 in 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: osdmap e314: 8 total, 8 up, 8 in 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[41707]: osdmap e315: 8 total, 8 up, 8 in 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: osdmap e314: 8 total, 8 up, 8 in 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]: dispatch 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3839227348' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi121-81537-58"}]': finished 2024-02-10T06:04:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:07 smithi121 ceph-mon[30020]: osdmap e315: 8 total, 8 up, 8 in 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[41707]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[30020]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:08 smithi146 ceph-mon[33795]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:09.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:09 smithi146 ceph-mon[33795]: osdmap e316: 8 total, 8 up, 8 in 2024-02-10T06:04:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[41707]: osdmap e316: 8 total, 8 up, 8 in 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[30020]: osdmap e316: 8 total, 8 up, 8 in 2024-02-10T06:04:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:10 smithi146 ceph-mon[33795]: pgmap v422: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:10 smithi146 ceph-mon[33795]: osdmap e317: 8 total, 8 up, 8 in 2024-02-10T06:04:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:10 smithi146 ceph-mon[33795]: osdmap e318: 8 total, 8 up, 8 in 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[41707]: pgmap v422: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[41707]: osdmap e317: 8 total, 8 up, 8 in 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[41707]: osdmap e318: 8 total, 8 up, 8 in 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[30020]: pgmap v422: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[30020]: osdmap e317: 8 total, 8 up, 8 in 2024-02-10T06:04:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi121-81537-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:10 smithi121 ceph-mon[30020]: osdmap e318: 8 total, 8 up, 8 in 2024-02-10T06:04:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[30020]: pgmap v425: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[30020]: osdmap e319: 8 total, 8 up, 8 in 2024-02-10T06:04:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[41707]: pgmap v425: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[41707]: osdmap e319: 8 total, 8 up, 8 in 2024-02-10T06:04:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:12 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:12 smithi146 ceph-mon[33795]: pgmap v425: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:12 smithi146 ceph-mon[33795]: osdmap e319: 8 total, 8 up, 8 in 2024-02-10T06:04:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:12 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:13 smithi121 ceph-mon[41707]: osdmap e320: 8 total, 8 up, 8 in 2024-02-10T06:04:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:13 smithi121 ceph-mon[30020]: osdmap e320: 8 total, 8 up, 8 in 2024-02-10T06:04:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:13 smithi146 ceph-mon[33795]: osdmap e320: 8 total, 8 up, 8 in 2024-02-10T06:04:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:14.449 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7462ddf6:::163:o.RoundTripAppendPP (3180 ms) 2024-02-10T06:04:14.449 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2772 ms) 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3010 ms) 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2997 ms) 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: setting pool EIO 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: max_success 99, min_failed 100 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4035 ms) 2024-02-10T06:04:14.450 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3009 ms) 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (127658 ms total) 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3008 ms) 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9269 ms) 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14703 ms) 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-10T06:04:14.451 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3010 ms) 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29991 ms total) 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3244 ms) 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3244 ms total) 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13958 ms) 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-10T06:04:14.452 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6773 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7034 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7012 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3013 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6936 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7014 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7021 ms) 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-10T06:04:14.453 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7016 ms) 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7044 ms) 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7014 ms) 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7029 ms) 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-10T06:04:14.454 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7049 ms) 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[41707]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[41707]: osdmap e321: 8 total, 8 up, 8 in 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[30020]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[30020]: osdmap e321: 8 total, 8 up, 8 in 2024-02-10T06:04:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:14 smithi146 ceph-mon[33795]: pgmap v428: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:14 smithi146 ceph-mon[33795]: osdmap e321: 8 total, 8 up, 8 in 2024-02-10T06:04:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]: dispatch 2024-02-10T06:04:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: osdmap e322: 8 total, 8 up, 8 in 2024-02-10T06:04:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: osdmap e322: 8 total, 8 up, 8 in 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:15.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2417240131' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi121-81537-59"}]': finished 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: osdmap e322: 8 total, 8 up, 8 in 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]: dispatch 2024-02-10T06:04:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:15.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]: dispatch 2024-02-10T06:04:16.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]': finished 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-02-10T06:04:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]': finished 2024-02-10T06:04:16.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]': finished 2024-02-10T06:04:16.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: osdmap e323: 8 total, 8 up, 8 in 2024-02-10T06:04:16.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:16.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]': finished 2024-02-10T06:04:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: osdmap e323: 8 total, 8 up, 8 in 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.b", "id": [3, 7]}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.f", "id": [0, 1]}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.11", "id": [3, 1]}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1f", "id": [5, 1]}]': finished 2024-02-10T06:04:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "268.9", "id": [4, 3]}]': finished 2024-02-10T06:04:16.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-45"}]': finished 2024-02-10T06:04:16.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: osdmap e323: 8 total, 8 up, 8 in 2024-02-10T06:04:16.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:16.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[41707]: osdmap e324: 8 total, 8 up, 8 in 2024-02-10T06:04:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:17 smithi121 ceph-mon[30020]: osdmap e324: 8 total, 8 up, 8 in 2024-02-10T06:04:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1792752591' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi121-81537-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:17 smithi146 ceph-mon[33795]: osdmap e324: 8 total, 8 up, 8 in 2024-02-10T06:04:18.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: pgmap v434: 324 pgs: 32 creating+peering, 6 peering, 286 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:18.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:04:18.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: osdmap e325: 8 total, 8 up, 8 in 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: pgmap v434: 324 pgs: 32 creating+peering, 6 peering, 286 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: osdmap e325: 8 total, 8 up, 8 in 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:18.731 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: pgmap v434: 324 pgs: 32 creating+peering, 6 peering, 286 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:04:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-10T06:04:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: osdmap e325: 8 total, 8 up, 8 in 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: osdmap e326: 8 total, 8 up, 8 in 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: osdmap e326: 8 total, 8 up, 8 in 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:19 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi121-81537-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: osdmap e326: 8 total, 8 up, 8 in 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:19 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[30020]: pgmap v437: 292 pgs: 32 unknown, 5 peering, 255 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[30020]: osdmap e327: 8 total, 8 up, 8 in 2024-02-10T06:04:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[41707]: pgmap v437: 292 pgs: 32 unknown, 5 peering, 255 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[41707]: osdmap e327: 8 total, 8 up, 8 in 2024-02-10T06:04:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:20.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:20 smithi146 ceph-mon[33795]: pgmap v437: 292 pgs: 32 unknown, 5 peering, 255 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:20.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:20 smithi146 ceph-mon[33795]: osdmap e327: 8 total, 8 up, 8 in 2024-02-10T06:04:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[41707]: osdmap e328: 8 total, 8 up, 8 in 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[30020]: osdmap e328: 8 total, 8 up, 8 in 2024-02-10T06:04:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:21.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi121-81537-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:21 smithi146 ceph-mon[33795]: osdmap e328: 8 total, 8 up, 8 in 2024-02-10T06:04:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[41707]: pgmap v440: 300 pgs: 8 creating+peering, 1 unknown, 291 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[41707]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:04:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[41707]: osdmap e329: 8 total, 8 up, 8 in 2024-02-10T06:04:22.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]: dispatch 2024-02-10T06:04:22.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[30020]: pgmap v440: 300 pgs: 8 creating+peering, 1 unknown, 291 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:22.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[30020]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:04:22.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:22.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[30020]: osdmap e329: 8 total, 8 up, 8 in 2024-02-10T06:04:22.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]: dispatch 2024-02-10T06:04:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:22 smithi146 ceph-mon[33795]: pgmap v440: 300 pgs: 8 creating+peering, 1 unknown, 291 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:22 smithi146 ceph-mon[33795]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-10T06:04:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:22 smithi146 ceph-mon[33795]: osdmap e329: 8 total, 8 up, 8 in 2024-02-10T06:04:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]: dispatch 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]': finished 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: osdmap e330: 8 total, 8 up, 8 in 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: osdmap e331: 8 total, 8 up, 8 in 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]': finished 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: osdmap e330: 8 total, 8 up, 8 in 2024-02-10T06:04:23.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: osdmap e331: 8 total, 8 up, 8 in 2024-02-10T06:04:23.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:23.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-47", "mode": "writeback"}]': finished 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: osdmap e330: 8 total, 8 up, 8 in 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: osdmap e331: 8 total, 8 up, 8 in 2024-02-10T06:04:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:04:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: osdmap e332: 8 total, 8 up, 8 in 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: osdmap e332: 8 total, 8 up, 8 in 2024-02-10T06:04:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:24.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3323246278' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]: dispatch 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi121-81537-61"}]': finished 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: osdmap e332: 8 total, 8 up, 8 in 2024-02-10T06:04:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: osdmap e333: 8 total, 8 up, 8 in 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: osdmap e333: 8 total, 8 up, 8 in 2024-02-10T06:04:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:25.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: osdmap e333: 8 total, 8 up, 8 in 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:26.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:26 smithi121 ceph-mon[41707]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-02-10T06:04:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:04:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:26 smithi121 ceph-mon[30020]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-02-10T06:04:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:04:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:26 smithi146 ceph-mon[33795]: pgmap v446: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-02-10T06:04:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:04:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: osdmap e334: 8 total, 8 up, 8 in 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: osdmap e335: 8 total, 8 up, 8 in 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: osdmap e334: 8 total, 8 up, 8 in 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:28.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:04:28.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: osdmap e335: 8 total, 8 up, 8 in 2024-02-10T06:04:28.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:04:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: osdmap e334: 8 total, 8 up, 8 in 2024-02-10T06:04:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:04:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi121-81537-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:04:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: osdmap e335: 8 total, 8 up, 8 in 2024-02-10T06:04:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:04:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:04:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[30020]: osdmap e336: 8 total, 8 up, 8 in 2024-02-10T06:04:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:04:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[41707]: osdmap e336: 8 total, 8 up, 8 in 2024-02-10T06:04:29.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:04:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:29 smithi146 ceph-mon[33795]: osdmap e336: 8 total, 8 up, 8 in 2024-02-10T06:04:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: pgmap v452: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: osdmap e337: 8 total, 8 up, 8 in 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: pgmap v452: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: osdmap e337: 8 total, 8 up, 8 in 2024-02-10T06:04:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: pgmap v452: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: osdmap e337: 8 total, 8 up, 8 in 2024-02-10T06:04:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: osdmap e338: 8 total, 8 up, 8 in 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: osdmap e338: 8 total, 8 up, 8 in 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]': finished 2024-02-10T06:04:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: osdmap e338: 8 total, 8 up, 8 in 2024-02-10T06:04:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]: dispatch 2024-02-10T06:04:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:32.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-47"}]: dispatch 2024-02-10T06:04:32.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:32.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1204343978' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi121-81537-62"}]': finished 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: pgmap v455: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: osdmap e339: 8 total, 8 up, 8 in 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: osdmap e340: 8 total, 8 up, 8 in 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: pgmap v455: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: osdmap e339: 8 total, 8 up, 8 in 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: osdmap e340: 8 total, 8 up, 8 in 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: pgmap v455: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: osdmap e339: 8 total, 8 up, 8 in 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi121-81537-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: osdmap e340: 8 total, 8 up, 8 in 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:33.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[41707]: pgmap v458: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:04:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[41707]: osdmap e341: 8 total, 8 up, 8 in 2024-02-10T06:04:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[30020]: pgmap v458: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:04:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[30020]: osdmap e341: 8 total, 8 up, 8 in 2024-02-10T06:04:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:34.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:34 smithi146 ceph-mon[33795]: pgmap v458: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:04:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:34 smithi146 ceph-mon[33795]: osdmap e341: 8 total, 8 up, 8 in 2024-02-10T06:04:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[41707]: osdmap e342: 8 total, 8 up, 8 in 2024-02-10T06:04:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[30020]: osdmap e342: 8 total, 8 up, 8 in 2024-02-10T06:04:35.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:35.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi121-81537-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:35 smithi146 ceph-mon[33795]: osdmap e342: 8 total, 8 up, 8 in 2024-02-10T06:04:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:36.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[41707]: pgmap v461: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[41707]: osdmap e343: 8 total, 8 up, 8 in 2024-02-10T06:04:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]: dispatch 2024-02-10T06:04:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[30020]: pgmap v461: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[30020]: osdmap e343: 8 total, 8 up, 8 in 2024-02-10T06:04:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]: dispatch 2024-02-10T06:04:36.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:36 smithi146 ceph-mon[33795]: pgmap v461: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:36 smithi146 ceph-mon[33795]: osdmap e343: 8 total, 8 up, 8 in 2024-02-10T06:04:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]: dispatch 2024-02-10T06:04:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]': finished 2024-02-10T06:04:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: osdmap e344: 8 total, 8 up, 8 in 2024-02-10T06:04:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]': finished 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: osdmap e344: 8 total, 8 up, 8 in 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:37.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-49", "mode": "readproxy"}]': finished 2024-02-10T06:04:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: osdmap e344: 8 total, 8 up, 8 in 2024-02-10T06:04:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: osdmap e345: 8 total, 8 up, 8 in 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: osdmap e345: 8 total, 8 up, 8 in 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: pgmap v464: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: osdmap e345: 8 total, 8 up, 8 in 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1210851462' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: osdmap e346: 8 total, 8 up, 8 in 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: osdmap e346: 8 total, 8 up, 8 in 2024-02-10T06:04:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi121-81537-63"}]': finished 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: osdmap e346: 8 total, 8 up, 8 in 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:40.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:40.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[30020]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[30020]: osdmap e347: 8 total, 8 up, 8 in 2024-02-10T06:04:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[41707]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[41707]: osdmap e347: 8 total, 8 up, 8 in 2024-02-10T06:04:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:41.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:40 smithi146 ceph-mon[33795]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:04:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi121-81537-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:40 smithi146 ceph-mon[33795]: osdmap e347: 8 total, 8 up, 8 in 2024-02-10T06:04:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:42.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:41 smithi146 ceph-mon[33795]: osdmap e348: 8 total, 8 up, 8 in 2024-02-10T06:04:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:41 smithi146 ceph-mon[33795]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:41 smithi146 ceph-mon[33795]: osdmap e349: 8 total, 8 up, 8 in 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[30020]: osdmap e348: 8 total, 8 up, 8 in 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[30020]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[30020]: osdmap e349: 8 total, 8 up, 8 in 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[41707]: osdmap e348: 8 total, 8 up, 8 in 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[41707]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi121-81537-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:42.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:41 smithi121 ceph-mon[41707]: osdmap e349: 8 total, 8 up, 8 in 2024-02-10T06:04:43.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:43 smithi146 ceph-mon[33795]: osdmap e350: 8 total, 8 up, 8 in 2024-02-10T06:04:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:43 smithi121 ceph-mon[30020]: osdmap e350: 8 total, 8 up, 8 in 2024-02-10T06:04:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:43 smithi121 ceph-mon[41707]: osdmap e350: 8 total, 8 up, 8 in 2024-02-10T06:04:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:44 smithi146 ceph-mon[33795]: pgmap v473: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:44 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:44 smithi146 ceph-mon[33795]: osdmap e351: 8 total, 8 up, 8 in 2024-02-10T06:04:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[41707]: pgmap v473: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[41707]: osdmap e351: 8 total, 8 up, 8 in 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[30020]: pgmap v473: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:04:45.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[30020]: osdmap e351: 8 total, 8 up, 8 in 2024-02-10T06:04:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:45.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:45.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:45 smithi146 ceph-mon[33795]: osdmap e352: 8 total, 8 up, 8 in 2024-02-10T06:04:45.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:45.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[41707]: osdmap e352: 8 total, 8 up, 8 in 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[30020]: osdmap e352: 8 total, 8 up, 8 in 2024-02-10T06:04:46.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1740401527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:46.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]: dispatch 2024-02-10T06:04:46.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:46.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: pgmap v476: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: osdmap e353: 8 total, 8 up, 8 in 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:04:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: pgmap v476: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: osdmap e353: 8 total, 8 up, 8 in 2024-02-10T06:04:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: pgmap v476: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi121-81537-64"}]': finished 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: osdmap e353: 8 total, 8 up, 8 in 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:04:47.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:04:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: osdmap e354: 8 total, 8 up, 8 in 2024-02-10T06:04:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: osdmap e354: 8 total, 8 up, 8 in 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi121-81537-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: osdmap e354: 8 total, 8 up, 8 in 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:48 smithi146 ceph-mon[33795]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:48 smithi146 ceph-mon[33795]: osdmap e355: 8 total, 8 up, 8 in 2024-02-10T06:04:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:48.979 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[41707]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:48.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:48.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[41707]: osdmap e355: 8 total, 8 up, 8 in 2024-02-10T06:04:48.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:48.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[30020]: pgmap v479: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:48.981 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:04:48.981 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[30020]: osdmap e355: 8 total, 8 up, 8 in 2024-02-10T06:04:48.981 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:49.637 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:5d165639:::164:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f43765fc:::165:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b4c720e9:::166:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e694b040:::167:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:afa38db2:::168:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:77ba9f53:::169:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:87495034:::170:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7c96bf0e:::171:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:dbe346cc:::172:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:e943ec24:::173:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f97a9c0c:::174:head 2024-02-10T06:04:49.638 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:6f26e74d:::175:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4f95e106:::176:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:0e6f2f8f:::177:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:05db05f1:::178:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:38a78d66:::179:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:d095610b:::180:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:a1a9d709:::181:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:1e5d39db:::182:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:f7df4fb9:::183:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:03a7f161:::184:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ba70721e:::185:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:28e5662d:::186:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:973d52de:::187:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:4303eb1c:::188:head 2024-02-10T06:04:49.639 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:b990b48e:::189:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:29b8165b:::190:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:3547f197:::191:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:7e260936:::192:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:1abec7b1:::193:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:10fdda93:::194:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:15817eea:::195:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:770bab57:::196:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:ed9e13e7:::197:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:71471a8f:::198:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: checking for 266:10fb1d02:::199:head 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7781 ms) 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,0 2024-02-10T06:04:49.640 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: first is 1707545044 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,1707545052,1707545053,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,1707545052,1707545053,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545044,1707545046,1707545047,1707545049,1707545050,1707545052,1707545053,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545047,1707545049,1707545050,1707545052,1707545053,1707545055,1707545056,0 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: first now 1707545047, trimmed 2024-02-10T06:04:49.641 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21105 ms) 2024-02-10T06:04:49.642 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-10T06:04:49.642 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: foo0 2024-02-10T06:04:49.642 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-10T06:04:49.642 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14057 ms) 2024-02-10T06:04:49.642 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-10T06:04:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: osdmap e356: 8 total, 8 up, 8 in 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:49 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:50.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:50.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:50.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:50.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: osdmap e356: 8 total, 8 up, 8 in 2024-02-10T06:04:50.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi121-81537-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]': finished 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: osdmap e356: 8 total, 8 up, 8 in 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-49"}]: dispatch 2024-02-10T06:04:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:49 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:50 smithi146 ceph-mon[33795]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:50 smithi146 ceph-mon[33795]: osdmap e357: 8 total, 8 up, 8 in 2024-02-10T06:04:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:50 smithi121 ceph-mon[41707]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:50 smithi121 ceph-mon[41707]: osdmap e357: 8 total, 8 up, 8 in 2024-02-10T06:04:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:50 smithi121 ceph-mon[30020]: pgmap v482: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:50 smithi121 ceph-mon[30020]: osdmap e357: 8 total, 8 up, 8 in 2024-02-10T06:04:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:51 smithi146 ceph-mon[33795]: osdmap e358: 8 total, 8 up, 8 in 2024-02-10T06:04:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[30020]: osdmap e358: 8 total, 8 up, 8 in 2024-02-10T06:04:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[41707]: osdmap e358: 8 total, 8 up, 8 in 2024-02-10T06:04:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:04:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: pgmap v485: 292 pgs: 31 creating+peering, 1 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: osdmap e359: 8 total, 8 up, 8 in 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: pgmap v485: 292 pgs: 31 creating+peering, 1 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: osdmap e359: 8 total, 8 up, 8 in 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: pgmap v485: 292 pgs: 31 creating+peering, 1 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: osdmap e359: 8 total, 8 up, 8 in 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/904350302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]: dispatch 2024-02-10T06:04:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:04:53.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: osdmap e360: 8 total, 8 up, 8 in 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: osdmap e360: 8 total, 8 up, 8 in 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi121-81537-65"}]': finished 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: osdmap e360: 8 total, 8 up, 8 in 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:04:55.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: osdmap e361: 8 total, 8 up, 8 in 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]: dispatch 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]': finished 2024-02-10T06:04:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:04:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:04:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:04:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: osdmap e361: 8 total, 8 up, 8 in 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]: dispatch 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]': finished 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: pgmap v488: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi121-81537-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: osdmap e361: 8 total, 8 up, 8 in 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]: dispatch 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:04:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-51", "mode": "writeback"}]': finished 2024-02-10T06:04:56.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: osdmap e362: 8 total, 8 up, 8 in 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: pgmap v491: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: osdmap e363: 8 total, 8 up, 8 in 2024-02-10T06:04:56.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: osdmap e362: 8 total, 8 up, 8 in 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: pgmap v491: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: osdmap e363: 8 total, 8 up, 8 in 2024-02-10T06:04:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: osdmap e362: 8 total, 8 up, 8 in 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: pgmap v491: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi121-81537-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: osdmap e363: 8 total, 8 up, 8 in 2024-02-10T06:04:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:04:56.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:04:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:04:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:57 smithi146 ceph-mon[33795]: osdmap e364: 8 total, 8 up, 8 in 2024-02-10T06:04:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[41707]: osdmap e364: 8 total, 8 up, 8 in 2024-02-10T06:04:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:04:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:04:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[30020]: osdmap e364: 8 total, 8 up, 8 in 2024-02-10T06:04:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: pgmap v494: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: osdmap e365: 8 total, 8 up, 8 in 2024-02-10T06:04:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: pgmap v494: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: osdmap e365: 8 total, 8 up, 8 in 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: pgmap v494: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:04:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:04:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:04:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:04:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: osdmap e365: 8 total, 8 up, 8 in 2024-02-10T06:04:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:04:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ O 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3022 ms) 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7037 ms) 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7018 ms) 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-10T06:04:59.684 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7056 ms) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7032 ms) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7040 ms) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7025 ms) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139145 ms total) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (300038 ms total) 2024-02-10T06:04:59.685 INFO:tasks.workunit.client.0.smithi121.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: osdmap e366: 8 total, 8 up, 8 in 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:05:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: osdmap e366: 8 total, 8 up, 8 in 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:05:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:04:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: osdmap e366: 8 total, 8 up, 8 in 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]: dispatch 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-51","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:05:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:04:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2078179335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi121-81537-66"}]': finished 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[41707]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[41707]: osdmap e367: 8 total, 8 up, 8 in 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[30020]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:05:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[30020]: osdmap e367: 8 total, 8 up, 8 in 2024-02-10T06:05:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:00 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:00 smithi146 ceph-mon[33795]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-10T06:05:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:00 smithi146 ceph-mon[33795]: osdmap e367: 8 total, 8 up, 8 in 2024-02-10T06:05:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:05:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:05:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:00 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[41707]: pgmap v499: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:05:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[30020]: pgmap v499: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:02.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:02 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:05:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:02 smithi146 ceph-mon[33795]: pgmap v499: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:02 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:05:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:02 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:05:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:04 smithi121 ceph-mon[41707]: pgmap v500: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:04.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:04 smithi121 ceph-mon[30020]: pgmap v500: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:04.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:04 smithi146 ceph-mon[33795]: pgmap v500: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:05 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:05.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:05 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:05.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:05 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:06 smithi121 ceph-mon[41707]: pgmap v501: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:05:06.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:06 smithi121 ceph-mon[30020]: pgmap v501: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:05:06.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:06 smithi146 ceph-mon[33795]: pgmap v501: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 693 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:05:06.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:07.577 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:07.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:07.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:08.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:08 smithi121 ceph-mon[41707]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:08.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:08 smithi121 ceph-mon[30020]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:08 smithi146 ceph-mon[33795]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:10 smithi121 ceph-mon[41707]: pgmap v503: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:10.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:10.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:10 smithi121 ceph-mon[30020]: pgmap v503: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:10.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:10.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:10 smithi146 ceph-mon[33795]: pgmap v503: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:11.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:05:11.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[30020]: osdmap e368: 8 total, 8 up, 8 in 2024-02-10T06:05:11.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:05:11.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[41707]: osdmap e368: 8 total, 8 up, 8 in 2024-02-10T06:05:11.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:11.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:05:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:11 smithi146 ceph-mon[33795]: osdmap e368: 8 total, 8 up, 8 in 2024-02-10T06:05:11.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[30020]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[30020]: osdmap e369: 8 total, 8 up, 8 in 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[41707]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:05:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:05:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[41707]: osdmap e369: 8 total, 8 up, 8 in 2024-02-10T06:05:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:12.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:12 smithi146 ceph-mon[33795]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:05:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]': finished 2024-02-10T06:05:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:12 smithi146 ceph-mon[33795]: osdmap e369: 8 total, 8 up, 8 in 2024-02-10T06:05:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-51"}]: dispatch 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[41707]: osdmap e370: 8 total, 8 up, 8 in 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[41707]: osdmap e371: 8 total, 8 up, 8 in 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[30020]: osdmap e370: 8 total, 8 up, 8 in 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[30020]: osdmap e371: 8 total, 8 up, 8 in 2024-02-10T06:05:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:13.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:13 smithi146 ceph-mon[33795]: osdmap e370: 8 total, 8 up, 8 in 2024-02-10T06:05:13.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:13 smithi146 ceph-mon[33795]: osdmap e371: 8 total, 8 up, 8 in 2024-02-10T06:05:13.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: pgmap v508: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: osdmap e372: 8 total, 8 up, 8 in 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-53"}]: dispatch 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: pgmap v508: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: osdmap e372: 8 total, 8 up, 8 in 2024-02-10T06:05:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:14.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-53"}]: dispatch 2024-02-10T06:05:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: pgmap v508: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-02-10T06:05:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:05:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: osdmap e372: 8 total, 8 up, 8 in 2024-02-10T06:05:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-53"}]: dispatch 2024-02-10T06:05:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[41707]: pgmap v511: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[41707]: osdmap e373: 8 total, 8 up, 8 in 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[30020]: pgmap v511: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[30020]: osdmap e373: 8 total, 8 up, 8 in 2024-02-10T06:05:16.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:16.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:16 smithi146 ceph-mon[33795]: pgmap v511: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:16 smithi146 ceph-mon[33795]: osdmap e373: 8 total, 8 up, 8 in 2024-02-10T06:05:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:16.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:17.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:17 smithi146 ceph-mon[33795]: osdmap e374: 8 total, 8 up, 8 in 2024-02-10T06:05:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[41707]: osdmap e374: 8 total, 8 up, 8 in 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[30020]: osdmap e374: 8 total, 8 up, 8 in 2024-02-10T06:05:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: pgmap v514: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: osdmap e375: 8 total, 8 up, 8 in 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-55"}]: dispatch 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: pgmap v514: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: osdmap e375: 8 total, 8 up, 8 in 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-55"}]: dispatch 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: pgmap v514: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: osdmap e375: 8 total, 8 up, 8 in 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-55"}]: dispatch 2024-02-10T06:05:19.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:19 smithi146 ceph-mon[33795]: osdmap e376: 8 total, 8 up, 8 in 2024-02-10T06:05:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:19 smithi146 ceph-mon[33795]: osdmap e377: 8 total, 8 up, 8 in 2024-02-10T06:05:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[30020]: osdmap e376: 8 total, 8 up, 8 in 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[30020]: osdmap e377: 8 total, 8 up, 8 in 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[41707]: osdmap e376: 8 total, 8 up, 8 in 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[41707]: osdmap e377: 8 total, 8 up, 8 in 2024-02-10T06:05:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:20.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:20 smithi146 ceph-mon[33795]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:20.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:20 smithi146 ceph-mon[33795]: osdmap e378: 8 total, 8 up, 8 in 2024-02-10T06:05:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-57"}]: dispatch 2024-02-10T06:05:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[30020]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[30020]: osdmap e378: 8 total, 8 up, 8 in 2024-02-10T06:05:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-57"}]: dispatch 2024-02-10T06:05:21.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[41707]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:21.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:21.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[41707]: osdmap e378: 8 total, 8 up, 8 in 2024-02-10T06:05:21.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:21.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-57"}]: dispatch 2024-02-10T06:05:22.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:22 smithi146 ceph-mon[33795]: pgmap v520: 292 pgs: 6 unknown, 286 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-10T06:05:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:22 smithi146 ceph-mon[33795]: osdmap e379: 8 total, 8 up, 8 in 2024-02-10T06:05:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:22 smithi121 ceph-mon[41707]: pgmap v520: 292 pgs: 6 unknown, 286 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-10T06:05:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:22 smithi121 ceph-mon[41707]: osdmap e379: 8 total, 8 up, 8 in 2024-02-10T06:05:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:22 smithi121 ceph-mon[30020]: pgmap v520: 292 pgs: 6 unknown, 286 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-02-10T06:05:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:22 smithi121 ceph-mon[30020]: osdmap e379: 8 total, 8 up, 8 in 2024-02-10T06:05:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:23 smithi146 ceph-mon[33795]: osdmap e380: 8 total, 8 up, 8 in 2024-02-10T06:05:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:23.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:23 smithi121 ceph-mon[41707]: osdmap e380: 8 total, 8 up, 8 in 2024-02-10T06:05:23.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:23 smithi121 ceph-mon[30020]: osdmap e380: 8 total, 8 up, 8 in 2024-02-10T06:05:23.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:24.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: osdmap e381: 8 total, 8 up, 8 in 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-59"}]: dispatch 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: osdmap e381: 8 total, 8 up, 8 in 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-59"}]: dispatch 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: osdmap e381: 8 total, 8 up, 8 in 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-59"}]: dispatch 2024-02-10T06:05:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:25.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:25 smithi146 ceph-mon[33795]: osdmap e382: 8 total, 8 up, 8 in 2024-02-10T06:05:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:25 smithi121 ceph-mon[41707]: osdmap e382: 8 total, 8 up, 8 in 2024-02-10T06:05:25.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:25 smithi121 ceph-mon[30020]: osdmap e382: 8 total, 8 up, 8 in 2024-02-10T06:05:26.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:26 smithi146 ceph-mon[33795]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:26 smithi146 ceph-mon[33795]: osdmap e383: 8 total, 8 up, 8 in 2024-02-10T06:05:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:26.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[41707]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[41707]: osdmap e383: 8 total, 8 up, 8 in 2024-02-10T06:05:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[30020]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:05:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[30020]: osdmap e383: 8 total, 8 up, 8 in 2024-02-10T06:05:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:27 smithi146 ceph-mon[33795]: osdmap e384: 8 total, 8 up, 8 in 2024-02-10T06:05:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[41707]: osdmap e384: 8 total, 8 up, 8 in 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[30020]: osdmap e384: 8 total, 8 up, 8 in 2024-02-10T06:05:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:28 smithi146 ceph-mon[33795]: pgmap v529: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:28 smithi146 ceph-mon[33795]: osdmap e385: 8 total, 8 up, 8 in 2024-02-10T06:05:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-61"}]: dispatch 2024-02-10T06:05:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[30020]: pgmap v529: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[30020]: osdmap e385: 8 total, 8 up, 8 in 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-61"}]: dispatch 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[41707]: pgmap v529: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[41707]: osdmap e385: 8 total, 8 up, 8 in 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:05:29.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-61"}]: dispatch 2024-02-10T06:05:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:29 smithi146 ceph-mon[33795]: osdmap e386: 8 total, 8 up, 8 in 2024-02-10T06:05:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:29 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:29 smithi146 ceph-mon[33795]: osdmap e387: 8 total, 8 up, 8 in 2024-02-10T06:05:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:30.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[30020]: osdmap e386: 8 total, 8 up, 8 in 2024-02-10T06:05:30.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:30.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[30020]: osdmap e387: 8 total, 8 up, 8 in 2024-02-10T06:05:30.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:30.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[41707]: osdmap e386: 8 total, 8 up, 8 in 2024-02-10T06:05:30.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:05:30.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[41707]: osdmap e387: 8 total, 8 up, 8 in 2024-02-10T06:05:30.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:05:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:30 smithi146 ceph-mon[33795]: pgmap v532: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:30 smithi146 ceph-mon[33795]: osdmap e388: 8 total, 8 up, 8 in 2024-02-10T06:05:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[30020]: pgmap v532: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[30020]: osdmap e388: 8 total, 8 up, 8 in 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[41707]: pgmap v532: 260 pgs: 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:05:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:05:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[41707]: osdmap e388: 8 total, 8 up, 8 in 2024-02-10T06:05:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:05:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:05:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:31 smithi146 ceph-mon[33795]: osdmap e389: 8 total, 8 up, 8 in 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[41707]: osdmap e389: 8 total, 8 up, 8 in 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:05:32.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:31 smithi121 ceph-mon[30020]: osdmap e389: 8 total, 8 up, 8 in 2024-02-10T06:05:32.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:32 smithi146 ceph-mon[33795]: pgmap v535: 292 pgs: 5 unknown, 287 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:05:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:32 smithi146 ceph-mon[33795]: osdmap e390: 8 total, 8 up, 8 in 2024-02-10T06:05:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:32 smithi121 ceph-mon[41707]: pgmap v535: 292 pgs: 5 unknown, 287 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:05:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:32 smithi121 ceph-mon[41707]: osdmap e390: 8 total, 8 up, 8 in 2024-02-10T06:05:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:32 smithi121 ceph-mon[30020]: pgmap v535: 292 pgs: 5 unknown, 287 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:05:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:32 smithi121 ceph-mon[30020]: osdmap e390: 8 total, 8 up, 8 in 2024-02-10T06:05:34.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:34 smithi146 ceph-mon[33795]: pgmap v538: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:34.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:34 smithi146 ceph-mon[33795]: osdmap e391: 8 total, 8 up, 8 in 2024-02-10T06:05:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:34 smithi121 ceph-mon[41707]: pgmap v538: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:34 smithi121 ceph-mon[41707]: osdmap e391: 8 total, 8 up, 8 in 2024-02-10T06:05:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:34 smithi121 ceph-mon[30020]: pgmap v538: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:05:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:34 smithi121 ceph-mon[30020]: osdmap e391: 8 total, 8 up, 8 in 2024-02-10T06:05:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:35.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:35 smithi146 ceph-mon[33795]: osdmap e392: 8 total, 8 up, 8 in 2024-02-10T06:05:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:35 smithi121 ceph-mon[41707]: osdmap e392: 8 total, 8 up, 8 in 2024-02-10T06:05:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:35 smithi121 ceph-mon[30020]: osdmap e392: 8 total, 8 up, 8 in 2024-02-10T06:05:36.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:36 smithi146 ceph-mon[33795]: pgmap v541: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:05:36.573 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:36 smithi146 ceph-mon[33795]: osdmap e393: 8 total, 8 up, 8 in 2024-02-10T06:05:36.573 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:36 smithi121 ceph-mon[41707]: pgmap v541: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:05:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:36 smithi121 ceph-mon[41707]: osdmap e393: 8 total, 8 up, 8 in 2024-02-10T06:05:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:36 smithi121 ceph-mon[30020]: pgmap v541: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-02-10T06:05:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:36 smithi121 ceph-mon[30020]: osdmap e393: 8 total, 8 up, 8 in 2024-02-10T06:05:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:38 smithi121 ceph-mon[30020]: pgmap v543: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-02-10T06:05:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:38 smithi121 ceph-mon[41707]: pgmap v543: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-02-10T06:05:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:38 smithi146 ceph-mon[33795]: pgmap v543: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-02-10T06:05:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:40 smithi121 ceph-mon[41707]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T06:05:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:40 smithi121 ceph-mon[30020]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T06:05:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:40 smithi146 ceph-mon[33795]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-10T06:05:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:42 smithi121 ceph-mon[41707]: pgmap v545: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:05:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:42 smithi121 ceph-mon[30020]: pgmap v545: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:05:42.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:42 smithi146 ceph-mon[33795]: pgmap v545: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:05:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:44 smithi121 ceph-mon[41707]: pgmap v546: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:05:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:44 smithi121 ceph-mon[41707]: osdmap e394: 8 total, 8 up, 8 in 2024-02-10T06:05:44.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:44 smithi121 ceph-mon[30020]: pgmap v546: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:05:44.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:44 smithi121 ceph-mon[30020]: osdmap e394: 8 total, 8 up, 8 in 2024-02-10T06:05:44.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:44 smithi146 ceph-mon[33795]: pgmap v546: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:05:44.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:44 smithi146 ceph-mon[33795]: osdmap e394: 8 total, 8 up, 8 in 2024-02-10T06:05:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:46 smithi146 ceph-mon[33795]: pgmap v548: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T06:05:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:46 smithi146 ceph-mon[33795]: osdmap e395: 8 total, 8 up, 8 in 2024-02-10T06:05:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:46.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[30020]: pgmap v548: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T06:05:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[30020]: osdmap e395: 8 total, 8 up, 8 in 2024-02-10T06:05:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[41707]: pgmap v548: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-10T06:05:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[41707]: osdmap e395: 8 total, 8 up, 8 in 2024-02-10T06:05:46.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:05:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:48.728 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:48 smithi121 ceph-mon[41707]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:48.729 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:48 smithi121 ceph-mon[30020]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:48 smithi146 ceph-mon[33795]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:50 smithi121 ceph-mon[41707]: pgmap v551: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:50 smithi121 ceph-mon[41707]: osdmap e396: 8 total, 8 up, 8 in 2024-02-10T06:05:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:50 smithi121 ceph-mon[30020]: pgmap v551: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:50 smithi121 ceph-mon[30020]: osdmap e396: 8 total, 8 up, 8 in 2024-02-10T06:05:50.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:50 smithi146 ceph-mon[33795]: pgmap v551: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:50.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:50 smithi146 ceph-mon[33795]: osdmap e396: 8 total, 8 up, 8 in 2024-02-10T06:05:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:52 smithi121 ceph-mon[41707]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:05:52.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:52 smithi121 ceph-mon[30020]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:05:52.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:52 smithi146 ceph-mon[33795]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:05:54.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:54 smithi121 ceph-mon[41707]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:54.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:54 smithi121 ceph-mon[30020]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:54.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:54 smithi146 ceph-mon[33795]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:05:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:05:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:05:56.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:56 smithi121 ceph-mon[41707]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:56.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:56 smithi121 ceph-mon[30020]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:56.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:56 smithi146 ceph-mon[33795]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:05:56.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:05:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:05:57.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:57.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:57 smithi121 ceph-mon[41707]: osdmap e397: 8 total, 8 up, 8 in 2024-02-10T06:05:57.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:57.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:57 smithi121 ceph-mon[30020]: osdmap e397: 8 total, 8 up, 8 in 2024-02-10T06:05:57.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:05:57.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:57 smithi146 ceph-mon[33795]: osdmap e397: 8 total, 8 up, 8 in 2024-02-10T06:05:58.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:05:58 smithi121 ceph-mon[41707]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:05:58.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:05:58 smithi121 ceph-mon[30020]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:05:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:05:58 smithi146 ceph-mon[33795]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:06:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:00 smithi121 ceph-mon[41707]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:00 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:00 smithi121 ceph-mon[30020]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:00.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:00 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:00.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:00 smithi146 ceph-mon[33795]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:00.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:00 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:06:01.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:01.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:01.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:06:01.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:01.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:01.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:01.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:06:01.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:06:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:06:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:01 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:06:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:02 smithi121 ceph-mon[41707]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:02.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:02 smithi121 ceph-mon[30020]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:02 smithi146 ceph-mon[33795]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:04 smithi121 ceph-mon[41707]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:04 smithi121 ceph-mon[41707]: osdmap e398: 8 total, 8 up, 8 in 2024-02-10T06:06:04.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:04 smithi121 ceph-mon[30020]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:04 smithi121 ceph-mon[30020]: osdmap e398: 8 total, 8 up, 8 in 2024-02-10T06:06:04.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:04 smithi146 ceph-mon[33795]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:06:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:04 smithi146 ceph-mon[33795]: osdmap e398: 8 total, 8 up, 8 in 2024-02-10T06:06:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:06.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:06 smithi146 ceph-mon[33795]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T06:06:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-63"}]: dispatch 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[41707]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-63"}]: dispatch 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[30020]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-63"}]: dispatch 2024-02-10T06:06:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:07 smithi121 ceph-mon[30020]: osdmap e399: 8 total, 8 up, 8 in 2024-02-10T06:06:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:07 smithi121 ceph-mon[41707]: osdmap e399: 8 total, 8 up, 8 in 2024-02-10T06:06:07.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:07.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:07 smithi146 ceph-mon[33795]: osdmap e399: 8 total, 8 up, 8 in 2024-02-10T06:06:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[30020]: pgmap v564: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[30020]: osdmap e400: 8 total, 8 up, 8 in 2024-02-10T06:06:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[41707]: pgmap v564: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[41707]: osdmap e400: 8 total, 8 up, 8 in 2024-02-10T06:06:08.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:08 smithi146 ceph-mon[33795]: pgmap v564: 260 pgs: 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:08 smithi146 ceph-mon[33795]: osdmap e400: 8 total, 8 up, 8 in 2024-02-10T06:06:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[41707]: osdmap e401: 8 total, 8 up, 8 in 2024-02-10T06:06:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[30020]: osdmap e401: 8 total, 8 up, 8 in 2024-02-10T06:06:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:09.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:09 smithi146 ceph-mon[33795]: osdmap e401: 8 total, 8 up, 8 in 2024-02-10T06:06:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[41707]: pgmap v567: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[41707]: osdmap e402: 8 total, 8 up, 8 in 2024-02-10T06:06:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[30020]: pgmap v567: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:10 smithi121 ceph-mon[30020]: osdmap e402: 8 total, 8 up, 8 in 2024-02-10T06:06:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:10 smithi146 ceph-mon[33795]: pgmap v567: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:10 smithi146 ceph-mon[33795]: osdmap e402: 8 total, 8 up, 8 in 2024-02-10T06:06:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:11 smithi121 ceph-mon[41707]: osdmap e403: 8 total, 8 up, 8 in 2024-02-10T06:06:11.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:11 smithi121 ceph-mon[30020]: osdmap e403: 8 total, 8 up, 8 in 2024-02-10T06:06:11.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:11 smithi146 ceph-mon[33795]: osdmap e403: 8 total, 8 up, 8 in 2024-02-10T06:06:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:12 smithi121 ceph-mon[41707]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:12 smithi121 ceph-mon[41707]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T06:06:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:12 smithi121 ceph-mon[30020]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:12 smithi121 ceph-mon[30020]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T06:06:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:12 smithi146 ceph-mon[33795]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:12 smithi146 ceph-mon[33795]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T06:06:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:13 smithi121 ceph-mon[41707]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T06:06:13.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:13 smithi121 ceph-mon[30020]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T06:06:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:13 smithi146 ceph-mon[33795]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T06:06:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:14 smithi121 ceph-mon[41707]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:14 smithi121 ceph-mon[30020]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:14 smithi146 ceph-mon[33795]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-10T06:06:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:15.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:15.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:15.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]: dispatch 2024-02-10T06:06:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:16 smithi146 ceph-mon[33795]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:06:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]': finished 2024-02-10T06:06:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:16 smithi146 ceph-mon[33795]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T06:06:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:16.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[30020]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:06:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]': finished 2024-02-10T06:06:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[30020]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T06:06:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[41707]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-02-10T06:06:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.4", "id": [7, 3]}]': finished 2024-02-10T06:06:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[41707]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T06:06:16.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:17 smithi121 ceph-mon[41707]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T06:06:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:17 smithi121 ceph-mon[30020]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T06:06:17.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:17.950 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:17 smithi146 ceph-mon[33795]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T06:06:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:18 smithi121 ceph-mon[41707]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 369 B/s wr, 3 op/s 2024-02-10T06:06:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:18 smithi121 ceph-mon[30020]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 369 B/s wr, 3 op/s 2024-02-10T06:06:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:18 smithi146 ceph-mon[33795]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 369 B/s wr, 3 op/s 2024-02-10T06:06:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:20 smithi121 ceph-mon[41707]: pgmap v578: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 313 B/s wr, 3 op/s 2024-02-10T06:06:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:20 smithi121 ceph-mon[41707]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T06:06:20.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:20 smithi121 ceph-mon[30020]: pgmap v578: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 313 B/s wr, 3 op/s 2024-02-10T06:06:20.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:20 smithi121 ceph-mon[30020]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T06:06:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:20 smithi146 ceph-mon[33795]: pgmap v578: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 313 B/s wr, 3 op/s 2024-02-10T06:06:20.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:20 smithi146 ceph-mon[33795]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T06:06:22.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:22 smithi121 ceph-mon[41707]: pgmap v580: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-10T06:06:22.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:22 smithi121 ceph-mon[30020]: pgmap v580: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-10T06:06:22.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:22 smithi146 ceph-mon[33795]: pgmap v580: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-10T06:06:23.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:23.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-65"}]: dispatch 2024-02-10T06:06:23.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:23.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-65"}]: dispatch 2024-02-10T06:06:23.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:23.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-65"}]: dispatch 2024-02-10T06:06:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:24 smithi121 ceph-mon[41707]: pgmap v581: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 272 B/s wr, 3 op/s 2024-02-10T06:06:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:24 smithi121 ceph-mon[41707]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T06:06:24.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:24 smithi121 ceph-mon[30020]: pgmap v581: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 272 B/s wr, 3 op/s 2024-02-10T06:06:24.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:24 smithi121 ceph-mon[30020]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T06:06:24.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:24 smithi146 ceph-mon[33795]: pgmap v581: 292 pgs: 292 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 272 B/s wr, 3 op/s 2024-02-10T06:06:24.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:24 smithi146 ceph-mon[33795]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T06:06:25.322 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:25.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:25 smithi121 ceph-mon[41707]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T06:06:25.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:25.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:25 smithi121 ceph-mon[30020]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T06:06:25.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:25.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:25 smithi146 ceph-mon[33795]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T06:06:25.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[41707]: pgmap v584: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[41707]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[30020]: pgmap v584: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:26.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[30020]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T06:06:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:26.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:26.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-mon[33795]: pgmap v584: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:26.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-mon[33795]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T06:06:26.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:27.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:27 smithi146 ceph-mon[33795]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T06:06:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:27.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-67"}]: dispatch 2024-02-10T06:06:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[30020]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T06:06:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-67"}]: dispatch 2024-02-10T06:06:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[41707]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T06:06:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-67"}]: dispatch 2024-02-10T06:06:28.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:28 smithi146 ceph-mon[33795]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:28.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:28 smithi146 ceph-mon[33795]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T06:06:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:28 smithi121 ceph-mon[41707]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:28 smithi121 ceph-mon[41707]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T06:06:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:28 smithi121 ceph-mon[30020]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:28 smithi121 ceph-mon[30020]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T06:06:29.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:29 smithi146 ceph-mon[33795]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T06:06:29.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:29 smithi121 ceph-mon[41707]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T06:06:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:29 smithi121 ceph-mon[30020]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T06:06:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:30.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: pgmap v590: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:06:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T06:06:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:30.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-69"}]: dispatch 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: pgmap v590: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-69"}]: dispatch 2024-02-10T06:06:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: pgmap v590: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:06:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T06:06:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-69"}]: dispatch 2024-02-10T06:06:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:31 smithi146 ceph-mon[33795]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T06:06:31.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:31 smithi121 ceph-mon[30020]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T06:06:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:31 smithi121 ceph-mon[41707]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T06:06:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:32.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:32 smithi146 ceph-mon[33795]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:06:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:32 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:32 smithi146 ceph-mon[33795]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T06:06:32.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[41707]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[41707]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[30020]: pgmap v593: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:06:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[30020]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T06:06:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:33.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:33.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T06:06:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:33.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:33 smithi146 ceph-mon[33795]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[41707]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T06:06:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:06:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:06:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:33 smithi121 ceph-mon[30020]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T06:06:34.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:34 smithi146 ceph-mon[33795]: pgmap v596: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:06:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:34 smithi121 ceph-mon[30020]: pgmap v596: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:06:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:34 smithi121 ceph-mon[41707]: pgmap v596: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:06:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:35.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:35 smithi146 ceph-mon[33795]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T06:06:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:35 smithi121 ceph-mon[30020]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T06:06:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:35 smithi121 ceph-mon[41707]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T06:06:36.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:36 smithi146 ceph-mon[33795]: pgmap v599: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:36 smithi121 ceph-mon[41707]: pgmap v599: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:36 smithi121 ceph-mon[30020]: pgmap v599: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:38.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:38 smithi146 ceph-mon[33795]: pgmap v600: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-02-10T06:06:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:38 smithi121 ceph-mon[41707]: pgmap v600: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-02-10T06:06:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:38 smithi121 ceph-mon[30020]: pgmap v600: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-02-10T06:06:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:40 smithi146 ceph-mon[33795]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-02-10T06:06:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:40 smithi121 ceph-mon[41707]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-02-10T06:06:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:40 smithi121 ceph-mon[30020]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-02-10T06:06:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:42 smithi146 ceph-mon[33795]: pgmap v602: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-10T06:06:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:42 smithi121 ceph-mon[41707]: pgmap v602: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-10T06:06:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:42 smithi121 ceph-mon[30020]: pgmap v602: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-10T06:06:44.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:44 smithi146 ceph-mon[33795]: pgmap v603: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-02-10T06:06:44.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:44 smithi121 ceph-mon[30020]: pgmap v603: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-02-10T06:06:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:44 smithi121 ceph-mon[41707]: pgmap v603: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-02-10T06:06:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:46 smithi146 ceph-mon[33795]: pgmap v604: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:06:46.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:46 smithi121 ceph-mon[41707]: pgmap v604: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:06:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:46 smithi121 ceph-mon[30020]: pgmap v604: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:06:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:06:48.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:48 smithi121 ceph-mon[41707]: pgmap v605: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:06:48.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:48 smithi121 ceph-mon[30020]: pgmap v605: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:06:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:48 smithi146 ceph-mon[33795]: pgmap v605: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-02-10T06:06:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:50 smithi121 ceph-mon[30020]: pgmap v606: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:50 smithi121 ceph-mon[41707]: pgmap v606: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:50 smithi146 ceph-mon[33795]: pgmap v606: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:52 smithi121 ceph-mon[41707]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:52 smithi121 ceph-mon[30020]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:52 smithi146 ceph-mon[33795]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:54 smithi121 ceph-mon[30020]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:54 smithi121 ceph-mon[41707]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:54.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:54 smithi146 ceph-mon[33795]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:06:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:06:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:06:55.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:55.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-71"}]: dispatch 2024-02-10T06:06:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:55.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-71"}]: dispatch 2024-02-10T06:06:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:06:55.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-71"}]: dispatch 2024-02-10T06:06:56.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:56 smithi146 ceph-mon[33795]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:56.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:56 smithi146 ceph-mon[33795]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T06:06:56.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:06:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:06:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:56 smithi121 ceph-mon[41707]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:56 smithi121 ceph-mon[41707]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T06:06:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:56 smithi121 ceph-mon[30020]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:06:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:56 smithi121 ceph-mon[30020]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T06:06:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[41707]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T06:06:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[30020]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T06:06:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:06:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:57 smithi146 ceph-mon[33795]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T06:06:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:06:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[30020]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[30020]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T06:06:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[41707]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[41707]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T06:06:58.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:58 smithi146 ceph-mon[33795]: pgmap v612: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:06:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:58 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:06:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:06:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:58 smithi146 ceph-mon[33795]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T06:06:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:06:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:06:59 smithi121 ceph-mon[30020]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T06:06:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:06:59 smithi121 ceph-mon[41707]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T06:06:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:06:59 smithi146 ceph-mon[33795]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T06:07:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:00 smithi121 ceph-mon[41707]: pgmap v615: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:00 smithi121 ceph-mon[41707]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T06:07:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:00 smithi121 ceph-mon[30020]: pgmap v615: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:00 smithi121 ceph-mon[30020]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T06:07:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:00 smithi146 ceph-mon[33795]: pgmap v615: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:00 smithi146 ceph-mon[33795]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:01.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:07:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:01.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:07:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:07:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:07:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:01 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:07:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:02 smithi121 ceph-mon[41707]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T06:07:02.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:02 smithi121 ceph-mon[30020]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T06:07:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:02 smithi146 ceph-mon[33795]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-02-10T06:07:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:04 smithi121 ceph-mon[41707]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T06:07:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:04 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:04.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:04 smithi121 ceph-mon[30020]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T06:07:04.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:04 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:04 smithi146 ceph-mon[33795]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T06:07:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:04 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:06.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:06 smithi146 ceph-mon[33795]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 939 B/s wr, 3 op/s 2024-02-10T06:07:06.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:06 smithi121 ceph-mon[41707]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 939 B/s wr, 3 op/s 2024-02-10T06:07:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:06 smithi121 ceph-mon[30020]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 939 B/s wr, 3 op/s 2024-02-10T06:07:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:07.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:08 smithi121 ceph-mon[41707]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 831 B/s wr, 3 op/s 2024-02-10T06:07:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:08 smithi121 ceph-mon[30020]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 831 B/s wr, 3 op/s 2024-02-10T06:07:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:08 smithi146 ceph-mon[33795]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 831 B/s wr, 3 op/s 2024-02-10T06:07:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[41707]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-02-10T06:07:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-73"}]: dispatch 2024-02-10T06:07:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[30020]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-02-10T06:07:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-73"}]: dispatch 2024-02-10T06:07:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:10 smithi146 ceph-mon[33795]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-02-10T06:07:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-73"}]: dispatch 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[41707]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[41707]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[30020]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[30020]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T06:07:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:11 smithi146 ceph-mon[33795]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T06:07:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:11 smithi146 ceph-mon[33795]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T06:07:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[41707]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[41707]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[30020]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:07:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[30020]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T06:07:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:13.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:12 smithi146 ceph-mon[33795]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:07:13.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:12 smithi146 ceph-mon[33795]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T06:07:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:14 smithi121 ceph-mon[41707]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:14 smithi121 ceph-mon[41707]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T06:07:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:14 smithi121 ceph-mon[30020]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:14.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:14 smithi121 ceph-mon[30020]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T06:07:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:14 smithi146 ceph-mon[33795]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:14.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:14 smithi146 ceph-mon[33795]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T06:07:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[41707]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-75"}]: dispatch 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[30020]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-75"}]: dispatch 2024-02-10T06:07:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:15 smithi146 ceph-mon[33795]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T06:07:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-75"}]: dispatch 2024-02-10T06:07:16.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:16 smithi146 ceph-mon[33795]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:16 smithi146 ceph-mon[33795]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T06:07:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[41707]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[41707]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T06:07:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[30020]: pgmap v629: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 659 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[30020]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T06:07:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[41707]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T06:07:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[30020]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T06:07:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:17.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:17 smithi146 ceph-mon[33795]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T06:07:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[41707]: pgmap v632: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[41707]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[30020]: pgmap v632: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[30020]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T06:07:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:18.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:18 smithi146 ceph-mon[33795]: pgmap v632: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:18 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:18 smithi146 ceph-mon[33795]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T06:07:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[41707]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T06:07:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-77"}]: dispatch 2024-02-10T06:07:19.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[30020]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T06:07:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-77"}]: dispatch 2024-02-10T06:07:19.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:19 smithi146 ceph-mon[33795]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T06:07:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-77"}]: dispatch 2024-02-10T06:07:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:20 smithi121 ceph-mon[41707]: pgmap v635: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:20 smithi121 ceph-mon[41707]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T06:07:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:20 smithi121 ceph-mon[30020]: pgmap v635: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:20 smithi121 ceph-mon[30020]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T06:07:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:20 smithi146 ceph-mon[33795]: pgmap v635: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-10T06:07:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:20 smithi146 ceph-mon[33795]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[41707]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[41707]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[30020]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:21.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:21 smithi121 ceph-mon[30020]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T06:07:21.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:21 smithi146 ceph-mon[33795]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T06:07:21.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:21 smithi146 ceph-mon[33795]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[41707]: pgmap v638: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[41707]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[30020]: pgmap v638: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:22 smithi121 ceph-mon[30020]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T06:07:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:22 smithi146 ceph-mon[33795]: pgmap v638: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:22 smithi146 ceph-mon[33795]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: pgmap v641: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: 294.4 deep-scrub starts 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[30020]: 294.4 deep-scrub ok 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: pgmap v641: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: 294.4 deep-scrub starts 2024-02-10T06:07:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:24 smithi121 ceph-mon[41707]: 294.4 deep-scrub ok 2024-02-10T06:07:24.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: pgmap v641: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.4"}]: dispatch 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: 294.4 deep-scrub starts 2024-02-10T06:07:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:24 smithi146 ceph-mon[33795]: 294.4 deep-scrub ok 2024-02-10T06:07:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:26.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:26 smithi146 ceph-mon[33795]: pgmap v643: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 2 op/s 2024-02-10T06:07:26.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:26 smithi121 ceph-mon[41707]: pgmap v643: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 2 op/s 2024-02-10T06:07:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:26 smithi121 ceph-mon[30020]: pgmap v643: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 2 op/s 2024-02-10T06:07:27.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:27.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:28 smithi121 ceph-mon[30020]: pgmap v644: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-10T06:07:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:28 smithi121 ceph-mon[41707]: pgmap v644: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-10T06:07:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:28 smithi146 ceph-mon[33795]: pgmap v644: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-10T06:07:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:30 smithi121 ceph-mon[41707]: pgmap v645: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 678 B/s wr, 1 op/s 2024-02-10T06:07:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:30 smithi121 ceph-mon[30020]: pgmap v645: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 678 B/s wr, 1 op/s 2024-02-10T06:07:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:30 smithi146 ceph-mon[33795]: pgmap v645: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 678 B/s wr, 1 op/s 2024-02-10T06:07:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:32.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:32 smithi146 ceph-mon[33795]: pgmap v646: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 598 B/s wr, 2 op/s 2024-02-10T06:07:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:32 smithi121 ceph-mon[30020]: pgmap v646: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 598 B/s wr, 2 op/s 2024-02-10T06:07:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:32 smithi121 ceph-mon[41707]: pgmap v646: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 598 B/s wr, 2 op/s 2024-02-10T06:07:34.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:34 smithi146 ceph-mon[33795]: pgmap v647: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:07:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:34 smithi121 ceph-mon[41707]: pgmap v647: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:07:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:34 smithi121 ceph-mon[30020]: pgmap v647: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:07:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:36 smithi146 ceph-mon[33795]: pgmap v648: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 443 B/s wr, 1 op/s 2024-02-10T06:07:36.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:36 smithi121 ceph-mon[41707]: pgmap v648: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 443 B/s wr, 1 op/s 2024-02-10T06:07:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:36 smithi121 ceph-mon[30020]: pgmap v648: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 443 B/s wr, 1 op/s 2024-02-10T06:07:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:38.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:38 smithi146 ceph-mon[33795]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-02-10T06:07:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:38 smithi121 ceph-mon[41707]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-02-10T06:07:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:38 smithi121 ceph-mon[30020]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-02-10T06:07:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:40 smithi146 ceph-mon[33795]: pgmap v650: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:07:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:40 smithi121 ceph-mon[41707]: pgmap v650: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:07:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:40 smithi121 ceph-mon[30020]: pgmap v650: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:07:42.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:42 smithi146 ceph-mon[33795]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:42 smithi121 ceph-mon[41707]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:42 smithi121 ceph-mon[30020]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:43.921 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:07:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:07:43.595+0000 7f7a16b24700 -1 snap_mapper.add_oid found existing snaps mapped on 294:22014ae9:test-rados-api-smithi121-81810-80::foo:2, removing 2024-02-10T06:07:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-79"}]: dispatch 2024-02-10T06:07:43.922 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:07:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:07:43.595+0000 7fd4511b6700 -1 snap_mapper.add_oid found existing snaps mapped on 294:22014ae9:test-rados-api-smithi121-81810-80::foo:2, removing 2024-02-10T06:07:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:44.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-79"}]: dispatch 2024-02-10T06:07:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:44.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-79"}]: dispatch 2024-02-10T06:07:44.079 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:07:43 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T06:07:43.595+0000 7fe3fa538700 -1 snap_mapper.add_oid found existing snaps mapped on 294:22014ae9:test-rados-api-smithi121-81810-80::foo:2, removing 2024-02-10T06:07:45.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:44 smithi146 ceph-mon[33795]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:44 smithi121 ceph-mon[41707]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:44 smithi121 ceph-mon[30020]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:07:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:46.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:45 smithi146 ceph-mon[33795]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T06:07:46.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:45 smithi146 ceph-mon[33795]: pgmap v654: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-02-10T06:07:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:45 smithi146 ceph-mon[33795]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T06:07:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[41707]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[41707]: pgmap v654: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[41707]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[30020]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T06:07:46.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[30020]: pgmap v654: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-02-10T06:07:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[30020]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T06:07:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:46.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:48 smithi146 ceph-mon[33795]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:07:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:48 smithi146 ceph-mon[33795]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T06:07:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[41707]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[41707]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[30020]: pgmap v657: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:07:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[30020]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T06:07:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:07:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T06:07:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:07:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:07:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T06:07:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:50.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T06:07:50.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:07:50.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:07:50.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T06:07:50.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:50 smithi146 ceph-mon[33795]: pgmap v660: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:07:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:50 smithi146 ceph-mon[33795]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[41707]: pgmap v660: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[41707]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[30020]: pgmap v660: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:50 smithi121 ceph-mon[30020]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T06:07:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T06:07:52.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T06:07:52.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-10T06:07:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:52 smithi146 ceph-mon[33795]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T06:07:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:52 smithi146 ceph-mon[33795]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T06:07:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:52 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[30020]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[30020]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[41707]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-10T06:07:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[41707]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T06:07:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:52 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-10T06:07:53.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T06:07:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:53 smithi146 ceph-mon[33795]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T06:07:53.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T06:07:54.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[30020]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T06:07:54.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-10T06:07:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[41707]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T06:07:54.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:07:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:54 smithi146 ceph-mon[33795]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:54 smithi146 ceph-mon[33795]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T06:07:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-81"}]: dispatch 2024-02-10T06:07:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[41707]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[41707]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-81"}]: dispatch 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[30020]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[30020]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-81"}]: dispatch 2024-02-10T06:07:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:07:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:07:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:07:55.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:55 smithi146 ceph-mon[33795]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T06:07:56.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:55 smithi121 ceph-mon[30020]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T06:07:56.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:55 smithi121 ceph-mon[41707]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T06:07:56.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:07:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[41707]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[41707]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[30020]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[30020]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T06:07:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:57.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:56 smithi146 ceph-mon[33795]: pgmap v669: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:07:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:56 smithi146 ceph-mon[33795]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T06:07:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:07:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[41707]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T06:07:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[30020]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T06:07:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:58.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:07:58.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:07:58.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:57 smithi146 ceph-mon[33795]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T06:07:58.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:07:58.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[41707]: pgmap v672: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[41707]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[30020]: pgmap v672: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[30020]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T06:07:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:07:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:58 smithi146 ceph-mon[33795]: pgmap v672: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-02-10T06:07:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:07:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:58 smithi146 ceph-mon[33795]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T06:07:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[41707]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[41707]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[30020]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:07:59 smithi121 ceph-mon[30020]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T06:08:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:59 smithi146 ceph-mon[33795]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T06:08:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:07:59 smithi146 ceph-mon[33795]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T06:08:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[41707]: pgmap v675: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-02-10T06:08:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[41707]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T06:08:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[30020]: pgmap v675: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-02-10T06:08:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:00 smithi121 ceph-mon[30020]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T06:08:01.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:00 smithi146 ceph-mon[33795]: pgmap v675: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-02-10T06:08:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:00 smithi146 ceph-mon[33795]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T06:08:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:08:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[41707]: pgmap v678: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[41707]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[30020]: pgmap v678: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[30020]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T06:08:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:02 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:08:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:02 smithi146 ceph-mon[33795]: pgmap v678: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:02 smithi146 ceph-mon[33795]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T06:08:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:08:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:08:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:02 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[41707]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-83"}]: dispatch 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[30020]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-83"}]: dispatch 2024-02-10T06:08:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:03 smithi146 ceph-mon[33795]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T06:08:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-83"}]: dispatch 2024-02-10T06:08:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:04 smithi121 ceph-mon[41707]: pgmap v681: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:04 smithi121 ceph-mon[41707]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T06:08:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:04 smithi121 ceph-mon[30020]: pgmap v681: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:04 smithi121 ceph-mon[30020]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T06:08:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:04 smithi146 ceph-mon[33795]: pgmap v681: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:04 smithi146 ceph-mon[33795]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T06:08:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:06.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:05 smithi146 ceph-mon[33795]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T06:08:06.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:05 smithi121 ceph-mon[41707]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T06:08:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:05 smithi121 ceph-mon[30020]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T06:08:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:06.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: pgmap v684: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: pgmap v684: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: pgmap v684: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T06:08:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[30020]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[41707]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T06:08:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:07 smithi146 ceph-mon[33795]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T06:08:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[30020]: pgmap v687: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[30020]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[41707]: pgmap v687: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[41707]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T06:08:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:08 smithi146 ceph-mon[33795]: pgmap v687: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-02-10T06:08:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_tier","val": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:08:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:08 smithi146 ceph-mon[33795]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T06:08:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[30020]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T06:08:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[41707]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T06:08:10.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:09 smithi146 ceph-mon[33795]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T06:08:10.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[41707]: pgmap v690: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[41707]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[30020]: pgmap v690: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:10 smithi121 ceph-mon[30020]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T06:08:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:10 smithi146 ceph-mon[33795]: pgmap v690: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-02-10T06:08:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:10 smithi146 ceph-mon[33795]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T06:08:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:11 smithi121 ceph-mon[30020]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T06:08:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:11 smithi121 ceph-mon[41707]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T06:08:12.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:11 smithi146 ceph-mon[33795]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[30020]: pgmap v693: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[30020]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-85"}]: dispatch 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[41707]: pgmap v693: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-10T06:08:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[41707]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T06:08:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-85"}]: dispatch 2024-02-10T06:08:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:13 smithi146 ceph-mon[33795]: pgmap v693: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-10T06:08:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:13 smithi146 ceph-mon[33795]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T06:08:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-85"}]: dispatch 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[41707]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[41707]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[41707]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[30020]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T06:08:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[30020]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:08:14.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[30020]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T06:08:14.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:14.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:14 smithi146 ceph-mon[33795]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T06:08:14.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:14 smithi146 ceph-mon[33795]: pgmap v696: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:08:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:14 smithi146 ceph-mon[33795]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T06:08:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout:K ] LibRadosTwoPoolsPP.ProxyRead (18112 ms) 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22679 ms) 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3008 ms) 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.807 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3020 ms) 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3002 ms) 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3024 ms) 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4028 ms) 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.808 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38103 ms) 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16811 ms) 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4028 ms) 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3018 ms) 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.809 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25094 ms) 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15054 ms) 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5021 ms) 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4026 ms) 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: waiting for scrubs... 2024-02-10T06:08:14.810 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: done waiting 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25055 ms) 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10052 ms) 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9061 ms) 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9063 ms) 2024-02-10T06:08:14.811 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-10T06:08:15.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:15.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:15.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[30020]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T06:08:15.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[41707]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T06:08:15.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:15 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:15 smithi146 ceph-mon[33795]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T06:08:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[41707]: pgmap v699: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[41707]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[30020]: pgmap v699: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[30020]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T06:08:16.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:16.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:16 smithi146 ceph-mon[33795]: pgmap v699: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:16 smithi146 ceph-mon[33795]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T06:08:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:16.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[41707]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]: dispatch 2024-02-10T06:08:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[30020]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T06:08:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]: dispatch 2024-02-10T06:08:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:17 smithi146 ceph-mon[33795]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T06:08:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]: dispatch 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[41707]: pgmap v702: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]': finished 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[41707]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[30020]: pgmap v702: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]': finished 2024-02-10T06:08:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[30020]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T06:08:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:18 smithi146 ceph-mon[33795]: pgmap v702: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_tier","val": "test-rados-api-smithi121-81810-89-test-flush"}]': finished 2024-02-10T06:08:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:18 smithi146 ceph-mon[33795]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T06:08:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:08:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[41707]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T06:08:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[41707]: pgmap v705: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[30020]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T06:08:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[30020]: pgmap v705: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:19 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:20.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:08:20.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:19 smithi146 ceph-mon[33795]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T06:08:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:08:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:19 smithi146 ceph-mon[33795]: pgmap v705: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-02-10T06:08:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:19 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:20 smithi121 ceph-mon[41707]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T06:08:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:20 smithi121 ceph-mon[30020]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T06:08:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:08:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:20 smithi146 ceph-mon[33795]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[41707]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-87"}]: dispatch 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[41707]: pgmap v708: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[30020]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-87"}]: dispatch 2024-02-10T06:08:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:21 smithi121 ceph-mon[30020]: pgmap v708: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:21 smithi146 ceph-mon[33795]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T06:08:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-87"}]: dispatch 2024-02-10T06:08:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:21 smithi146 ceph-mon[33795]: pgmap v708: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:08:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:22 smithi121 ceph-mon[41707]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T06:08:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:22 smithi121 ceph-mon[30020]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T06:08:23.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:22 smithi146 ceph-mon[33795]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T06:08:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:23 smithi146 ceph-mon[33795]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T06:08:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:23 smithi146 ceph-mon[33795]: pgmap v711: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:08:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[41707]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T06:08:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[41707]: pgmap v711: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:08:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[30020]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T06:08:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:23 smithi121 ceph-mon[30020]: pgmap v711: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:08:25.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:24 smithi146 ceph-mon[33795]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T06:08:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:25.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:25.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[41707]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T06:08:25.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[30020]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T06:08:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:08:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:25 smithi146 ceph-mon[33795]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T06:08:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:25 smithi146 ceph-mon[33795]: pgmap v714: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[30020]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T06:08:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[30020]: pgmap v714: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:08:26.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[41707]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T06:08:26.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:25 smithi121 ceph-mon[41707]: pgmap v714: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:26.849 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:26 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:26 smithi146 ceph-mon[33795]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T06:08:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:26 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[41707]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T06:08:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[30020]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T06:08:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:26 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:27 smithi146 ceph-mon[33795]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T06:08:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-90"}]: dispatch 2024-02-10T06:08:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:27 smithi146 ceph-mon[33795]: pgmap v717: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[41707]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-90"}]: dispatch 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[41707]: pgmap v717: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[30020]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-90"}]: dispatch 2024-02-10T06:08:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:27 smithi121 ceph-mon[30020]: pgmap v717: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-02-10T06:08:29.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:28 smithi146 ceph-mon[33795]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T06:08:29.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:28 smithi121 ceph-mon[30020]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T06:08:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:28 smithi121 ceph-mon[41707]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T06:08:30.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:29 smithi146 ceph-mon[33795]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T06:08:30.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:30.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:29 smithi146 ceph-mon[33795]: pgmap v720: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:30.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:30.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:29 smithi146 ceph-mon[33795]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T06:08:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[30020]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T06:08:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[30020]: pgmap v720: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:30.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[30020]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T06:08:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[41707]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T06:08:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[41707]: pgmap v720: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:30.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:29 smithi121 ceph-mon[41707]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T06:08:31.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:31.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:32.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:31 smithi146 ceph-mon[33795]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T06:08:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:31 smithi146 ceph-mon[33795]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[41707]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[41707]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[30020]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[30020]: pgmap v723: 292 pgs: 292 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:32.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:32 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:32 smithi146 ceph-mon[33795]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T06:08:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-92"}]: dispatch 2024-02-10T06:08:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[41707]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T06:08:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-92"}]: dispatch 2024-02-10T06:08:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:08:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[30020]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T06:08:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:33.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-92"}]: dispatch 2024-02-10T06:08:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:33 smithi146 ceph-mon[33795]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T06:08:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:33 smithi146 ceph-mon[33795]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:33 smithi121 ceph-mon[30020]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T06:08:34.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:33 smithi121 ceph-mon[30020]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:33 smithi121 ceph-mon[41707]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T06:08:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:33 smithi121 ceph-mon[41707]: pgmap v726: 260 pgs: 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:08:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:34 smithi146 ceph-mon[33795]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T06:08:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:34 smithi121 ceph-mon[30020]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T06:08:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:34 smithi121 ceph-mon[41707]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T06:08:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:08:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[41707]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T06:08:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[41707]: pgmap v729: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[30020]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T06:08:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:35 smithi121 ceph-mon[30020]: pgmap v729: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:08:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:35 smithi146 ceph-mon[33795]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T06:08:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:08:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:35 smithi146 ceph-mon[33795]: pgmap v729: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:08:36.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:36 smithi121 ceph-mon[41707]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T06:08:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:36 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:36 smithi121 ceph-mon[30020]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T06:08:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:36 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:37.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:36 smithi146 ceph-mon[33795]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T06:08:37.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:36 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:37 smithi121 ceph-mon[41707]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T06:08:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:37 smithi121 ceph-mon[41707]: pgmap v732: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:08:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:37 smithi121 ceph-mon[30020]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T06:08:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:37 smithi121 ceph-mon[30020]: pgmap v732: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:08:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:37 smithi146 ceph-mon[33795]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T06:08:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:37 smithi146 ceph-mon[33795]: pgmap v732: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-10T06:08:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:40 smithi146 ceph-mon[33795]: pgmap v733: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T06:08:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:40 smithi121 ceph-mon[41707]: pgmap v733: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T06:08:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:40 smithi121 ceph-mon[30020]: pgmap v733: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-02-10T06:08:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:42 smithi146 ceph-mon[33795]: pgmap v734: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-02-10T06:08:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:42 smithi121 ceph-mon[41707]: pgmap v734: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-02-10T06:08:42.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:42 smithi121 ceph-mon[30020]: pgmap v734: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-02-10T06:08:44.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:44 smithi146 ceph-mon[33795]: pgmap v735: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T06:08:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:44 smithi121 ceph-mon[41707]: pgmap v735: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T06:08:44.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:44 smithi121 ceph-mon[30020]: pgmap v735: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-02-10T06:08:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:46.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:46 smithi146 ceph-mon[33795]: pgmap v736: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1001 B/s wr, 4 op/s 2024-02-10T06:08:46.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:46 smithi121 ceph-mon[41707]: pgmap v736: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1001 B/s wr, 4 op/s 2024-02-10T06:08:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:46 smithi121 ceph-mon[30020]: pgmap v736: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1001 B/s wr, 4 op/s 2024-02-10T06:08:47.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:08:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:48 smithi146 ceph-mon[33795]: pgmap v737: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-02-10T06:08:48.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:48 smithi146 ceph-mon[33795]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T06:08:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:48 smithi121 ceph-mon[41707]: pgmap v737: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-02-10T06:08:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:48 smithi121 ceph-mon[41707]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T06:08:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:48 smithi121 ceph-mon[30020]: pgmap v737: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-02-10T06:08:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:48 smithi121 ceph-mon[30020]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T06:08:50.630 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:50 smithi121 ceph-mon[41707]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-02-10T06:08:50.631 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:50 smithi121 ceph-mon[30020]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-02-10T06:08:50.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:50 smithi146 ceph-mon[33795]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-02-10T06:08:52.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:52 smithi146 ceph-mon[33795]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:52 smithi121 ceph-mon[30020]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:52 smithi121 ceph-mon[41707]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:54 smithi146 ceph-mon[33795]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:54.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:54 smithi146 ceph-mon[33795]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T06:08:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:54 smithi121 ceph-mon[41707]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:54 smithi121 ceph-mon[41707]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T06:08:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:54 smithi121 ceph-mon[30020]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:54 smithi121 ceph-mon[30020]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T06:08:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:08:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:08:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:08:56.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:56 smithi146 ceph-mon[33795]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:56.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:08:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:08:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:56 smithi121 ceph-mon[41707]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:56 smithi121 ceph-mon[30020]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:57.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:08:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:58 smithi146 ceph-mon[33795]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-94"}]: dispatch 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[41707]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-94"}]: dispatch 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[30020]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:08:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-94"}]: dispatch 2024-02-10T06:08:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:59 smithi146 ceph-mon[33795]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T06:08:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:59 smithi146 ceph-mon[33795]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T06:08:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:08:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[30020]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T06:09:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[30020]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T06:09:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[41707]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T06:09:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[41707]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T06:09:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:08:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:00 smithi146 ceph-mon[33795]: pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:00 smithi146 ceph-mon[33795]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T06:09:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[41707]: pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[41707]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[30020]: pgmap v746: 260 pgs: 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[30020]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T06:09:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:02.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:02 smithi146 ceph-mon[33795]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:02.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:02 smithi146 ceph-mon[33795]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T06:09:02.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:02.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[41707]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[41707]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T06:09:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[30020]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[30020]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T06:09:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:09:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi146 to 3248M 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: Adjusting osd_memory_target on smithi121 to 3248M 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:03.674 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi146 to 3248M 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: Adjusting osd_memory_target on smithi121 to 3248M 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi146 to 3248M 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: Adjusting osd_memory_target on smithi121 to 3248M 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:09:03.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:04.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:04 smithi146 ceph-mon[33795]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T06:09:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:04 smithi121 ceph-mon[41707]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T06:09:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:04 smithi121 ceph-mon[30020]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-02-10T06:09:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:06.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:06 smithi146 ceph-mon[33795]: pgmap v752: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T06:09:06.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:06 smithi121 ceph-mon[41707]: pgmap v752: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T06:09:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:06 smithi121 ceph-mon[30020]: pgmap v752: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-02-10T06:09:07.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:08.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:08 smithi146 ceph-mon[33795]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 791 B/s wr, 2 op/s 2024-02-10T06:09:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:08 smithi121 ceph-mon[41707]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 791 B/s wr, 2 op/s 2024-02-10T06:09:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:08 smithi121 ceph-mon[30020]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 791 B/s wr, 2 op/s 2024-02-10T06:09:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:10 smithi146 ceph-mon[33795]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-02-10T06:09:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:10 smithi121 ceph-mon[41707]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-02-10T06:09:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:10 smithi121 ceph-mon[30020]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 701 B/s wr, 2 op/s 2024-02-10T06:09:11.671 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:09:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:09:11.404+0000 7f7a0db12700 -1 snap_mapper.add_oid found existing snaps mapped on 100:b955f034:test-rados-api-smithi121-81810-97::foo:21, removing 2024-02-10T06:09:11.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:09:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:09:11.404+0000 7fd4559bf700 -1 snap_mapper.add_oid found existing snaps mapped on 100:b955f034:test-rados-api-smithi121-81810-97::foo:21, removing 2024-02-10T06:09:11.673 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:09:11 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T06:09:11.404+0000 7f115eb6b700 -1 snap_mapper.add_oid found existing snaps mapped on 100:b955f034:test-rados-api-smithi121-81810-97::foo:21, removing 2024-02-10T06:09:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-96"}]: dispatch 2024-02-10T06:09:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-96"}]: dispatch 2024-02-10T06:09:12.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-96"}]: dispatch 2024-02-10T06:09:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:12 smithi121 ceph-mon[30020]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:09:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:12 smithi121 ceph-mon[30020]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T06:09:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:12 smithi121 ceph-mon[41707]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:09:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:12 smithi121 ceph-mon[41707]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T06:09:13.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:12 smithi146 ceph-mon[33795]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:09:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:12 smithi146 ceph-mon[33795]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T06:09:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:13 smithi146 ceph-mon[33795]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T06:09:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:13.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:09:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=cleanup t=2024-02-10T06:09:13.594608149Z level=info msg="Completed cleanup jobs" duration=3.216261ms 2024-02-10T06:09:13.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:09:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore.transactions t=2024-02-10T06:09:13.601973129Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T06:09:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:13 smithi121 ceph-mon[30020]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T06:09:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:13 smithi121 ceph-mon[41707]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T06:09:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[30020]: pgmap v758: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[30020]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[41707]: pgmap v758: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:15.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[41707]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T06:09:15.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:15.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:14 smithi146 ceph-mon[33795]: pgmap v758: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:14 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:14 smithi146 ceph-mon[33795]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T06:09:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[41707]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]: dispatch 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[30020]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T06:09:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]: dispatch 2024-02-10T06:09:16.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:15 smithi146 ceph-mon[33795]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T06:09:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]: dispatch 2024-02-10T06:09:16.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: pgmap v761: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]': finished 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: pgmap v761: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]': finished 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: pgmap v761: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 657 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-98", "mode": "writeback"}]': finished 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: osdmap e504: 8 total, 8 up, 8 in 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: osdmap e505: 8 total, 8 up, 8 in 2024-02-10T06:09:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: osdmap e504: 8 total, 8 up, 8 in 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: osdmap e505: 8 total, 8 up, 8 in 2024-02-10T06:09:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T06:09:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: osdmap e504: 8 total, 8 up, 8 in 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: osdmap e505: 8 total, 8 up, 8 in 2024-02-10T06:09:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]: dispatch 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[41707]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]': finished 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[41707]: osdmap e506: 8 total, 8 up, 8 in 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[30020]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]': finished 2024-02-10T06:09:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[30020]: osdmap e506: 8 total, 8 up, 8 in 2024-02-10T06:09:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:18 smithi146 ceph-mon[33795]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_count","val": "1"}]': finished 2024-02-10T06:09:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:18 smithi146 ceph-mon[33795]: osdmap e506: 8 total, 8 up, 8 in 2024-02-10T06:09:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[41707]: osdmap e507: 8 total, 8 up, 8 in 2024-02-10T06:09:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T06:09:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[30020]: osdmap e507: 8 total, 8 up, 8 in 2024-02-10T06:09:20.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T06:09:20.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:19 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:19 smithi146 ceph-mon[33795]: osdmap e507: 8 total, 8 up, 8 in 2024-02-10T06:09:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]: dispatch 2024-02-10T06:09:20.978 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[41707]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:20.979 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]': finished 2024-02-10T06:09:20.979 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[41707]: osdmap e508: 8 total, 8 up, 8 in 2024-02-10T06:09:20.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[30020]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:20.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]': finished 2024-02-10T06:09:20.979 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:20 smithi121 ceph-mon[30020]: osdmap e508: 8 total, 8 up, 8 in 2024-02-10T06:09:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:20 smithi146 ceph-mon[33795]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:09:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-98","var": "target_max_objects","val": "250"}]': finished 2024-02-10T06:09:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:20 smithi146 ceph-mon[33795]: osdmap e508: 8 total, 8 up, 8 in 2024-02-10T06:09:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:22.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[41707]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[41707]: osdmap e509: 8 total, 8 up, 8 in 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[30020]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[30020]: osdmap e509: 8 total, 8 up, 8 in 2024-02-10T06:09:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:22 smithi146 ceph-mon[33795]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:22 smithi146 ceph-mon[33795]: osdmap e509: 8 total, 8 up, 8 in 2024-02-10T06:09:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]: dispatch 2024-02-10T06:09:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:23 smithi121 ceph-mon[41707]: osdmap e510: 8 total, 8 up, 8 in 2024-02-10T06:09:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:23 smithi121 ceph-mon[30020]: osdmap e510: 8 total, 8 up, 8 in 2024-02-10T06:09:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-98"}]': finished 2024-02-10T06:09:24.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:23 smithi146 ceph-mon[33795]: osdmap e510: 8 total, 8 up, 8 in 2024-02-10T06:09:25.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:24 smithi121 ceph-mon[41707]: pgmap v773: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:25.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:24 smithi121 ceph-mon[41707]: osdmap e511: 8 total, 8 up, 8 in 2024-02-10T06:09:25.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:24 smithi121 ceph-mon[30020]: pgmap v773: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:25.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:24 smithi121 ceph-mon[30020]: osdmap e511: 8 total, 8 up, 8 in 2024-02-10T06:09:25.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:24 smithi146 ceph-mon[33795]: pgmap v773: 292 pgs: 292 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:24 smithi146 ceph-mon[33795]: osdmap e511: 8 total, 8 up, 8 in 2024-02-10T06:09:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:25 smithi121 ceph-mon[30020]: osdmap e512: 8 total, 8 up, 8 in 2024-02-10T06:09:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:25 smithi121 ceph-mon[41707]: osdmap e512: 8 total, 8 up, 8 in 2024-02-10T06:09:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:25 smithi146 ceph-mon[33795]: osdmap e512: 8 total, 8 up, 8 in 2024-02-10T06:09:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:26.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[41707]: pgmap v776: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[41707]: osdmap e513: 8 total, 8 up, 8 in 2024-02-10T06:09:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[30020]: pgmap v776: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[30020]: osdmap e513: 8 total, 8 up, 8 in 2024-02-10T06:09:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-mon[33795]: pgmap v776: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-mon[33795]: osdmap e513: 8 total, 8 up, 8 in 2024-02-10T06:09:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:09:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: osdmap e514: 8 total, 8 up, 8 in 2024-02-10T06:09:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: osdmap e515: 8 total, 8 up, 8 in 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]: dispatch 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: osdmap e514: 8 total, 8 up, 8 in 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: osdmap e515: 8 total, 8 up, 8 in 2024-02-10T06:09:28.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]: dispatch 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: osdmap e514: 8 total, 8 up, 8 in 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-6", "overlaypool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: osdmap e515: 8 total, 8 up, 8 in 2024-02-10T06:09:28.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]: dispatch 2024-02-10T06:09:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[41707]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]': finished 2024-02-10T06:09:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[41707]: osdmap e516: 8 total, 8 up, 8 in 2024-02-10T06:09:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:09:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[30020]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]': finished 2024-02-10T06:09:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[30020]: osdmap e516: 8 total, 8 up, 8 in 2024-02-10T06:09:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:09:29.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:28 smithi146 ceph-mon[33795]: pgmap v779: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:28 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-100", "mode": "writeback"}]': finished 2024-02-10T06:09:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:28 smithi146 ceph-mon[33795]: osdmap e516: 8 total, 8 up, 8 in 2024-02-10T06:09:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[41707]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[41707]: osdmap e517: 8 total, 8 up, 8 in 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[30020]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:09:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[30020]: osdmap e517: 8 total, 8 up, 8 in 2024-02-10T06:09:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:31.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:30 smithi146 ceph-mon[33795]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:09:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:30 smithi146 ceph-mon[33795]: osdmap e517: 8 total, 8 up, 8 in 2024-02-10T06:09:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: osdmap e518: 8 total, 8 up, 8 in 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: osdmap e519: 8 total, 8 up, 8 in 2024-02-10T06:09:32.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: osdmap e518: 8 total, 8 up, 8 in 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: osdmap e519: 8 total, 8 up, 8 in 2024-02-10T06:09:32.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: osdmap e518: 8 total, 8 up, 8 in 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: osdmap e519: 8 total, 8 up, 8 in 2024-02-10T06:09:32.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-10T06:09:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[30020]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T06:09:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[30020]: osdmap e520: 8 total, 8 up, 8 in 2024-02-10T06:09:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[41707]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T06:09:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:32 smithi121 ceph-mon[41707]: osdmap e520: 8 total, 8 up, 8 in 2024-02-10T06:09:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:32 smithi146 ceph-mon[33795]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-10T06:09:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:32 smithi146 ceph-mon[33795]: osdmap e520: 8 total, 8 up, 8 in 2024-02-10T06:09:34.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:34.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:33 smithi121 ceph-mon[41707]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:34.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:34.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:33 smithi121 ceph-mon[30020]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:34.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:33 smithi146 ceph-mon[33795]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:35.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:35.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:35.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[30020]: osdmap e521: 8 total, 8 up, 8 in 2024-02-10T06:09:35.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:35.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:35.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:35.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[41707]: osdmap e521: 8 total, 8 up, 8 in 2024-02-10T06:09:35.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:35.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:34 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]': finished 2024-02-10T06:09:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:34 smithi146 ceph-mon[33795]: osdmap e521: 8 total, 8 up, 8 in 2024-02-10T06:09:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]: dispatch 2024-02-10T06:09:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:34 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[41707]: osdmap e522: 8 total, 8 up, 8 in 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[41707]: pgmap v791: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[30020]: osdmap e522: 8 total, 8 up, 8 in 2024-02-10T06:09:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:35 smithi121 ceph-mon[30020]: pgmap v791: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:36.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-100"}]': finished 2024-02-10T06:09:36.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:35 smithi146 ceph-mon[33795]: osdmap e522: 8 total, 8 up, 8 in 2024-02-10T06:09:36.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:35 smithi146 ceph-mon[33795]: pgmap v791: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:36.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:36 smithi121 ceph-mon[41707]: osdmap e523: 8 total, 8 up, 8 in 2024-02-10T06:09:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:36 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:36 smithi121 ceph-mon[30020]: osdmap e523: 8 total, 8 up, 8 in 2024-02-10T06:09:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:36 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:36 smithi146 ceph-mon[33795]: osdmap e523: 8 total, 8 up, 8 in 2024-02-10T06:09:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:36 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:38.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:37 smithi146 ceph-mon[33795]: osdmap e524: 8 total, 8 up, 8 in 2024-02-10T06:09:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:37 smithi146 ceph-mon[33795]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[41707]: osdmap e524: 8 total, 8 up, 8 in 2024-02-10T06:09:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[41707]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:38.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[30020]: osdmap e524: 8 total, 8 up, 8 in 2024-02-10T06:09:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:37 smithi121 ceph-mon[30020]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:39.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:38 smithi146 ceph-mon[33795]: osdmap e525: 8 total, 8 up, 8 in 2024-02-10T06:09:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:09:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[30020]: osdmap e525: 8 total, 8 up, 8 in 2024-02-10T06:09:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:39.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:09:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[41707]: osdmap e525: 8 total, 8 up, 8 in 2024-02-10T06:09:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-10T06:09:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: osdmap e526: 8 total, 8 up, 8 in 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: pgmap v797: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: osdmap e527: 8 total, 8 up, 8 in 2024-02-10T06:09:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: osdmap e526: 8 total, 8 up, 8 in 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: pgmap v797: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: osdmap e527: 8 total, 8 up, 8 in 2024-02-10T06:09:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: osdmap e526: 8 total, 8 up, 8 in 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: pgmap v797: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: osdmap e527: 8 total, 8 up, 8 in 2024-02-10T06:09:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-10T06:09:42.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:09:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:41 smithi146 ceph-mon[33795]: osdmap e528: 8 total, 8 up, 8 in 2024-02-10T06:09:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-102"}]: dispatch 2024-02-10T06:09:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:41 smithi146 ceph-mon[33795]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:09:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:09:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[41707]: osdmap e528: 8 total, 8 up, 8 in 2024-02-10T06:09:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-102"}]: dispatch 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[41707]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[30020]: osdmap e528: 8 total, 8 up, 8 in 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-6"}]: dispatch 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/291424282' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-6", "tierpool": "test-rados-api-smithi121-81810-102"}]: dispatch 2024-02-10T06:09:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:41 smithi121 ceph-mon[30020]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-02-10T06:09:43.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:42 smithi146 ceph-mon[33795]: osdmap e529: 8 total, 8 up, 8 in 2024-02-10T06:09:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:42 smithi121 ceph-mon[41707]: osdmap e529: 8 total, 8 up, 8 in 2024-02-10T06:09:43.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:42 smithi121 ceph-mon[30020]: osdmap e529: 8 total, 8 up, 8 in 2024-02-10T06:09:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: osdmap e530: 8 total, 8 up, 8 in 2024-02-10T06:09:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:43 smithi146 ceph-mon[33795]: pgmap v803: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: osdmap e530: 8 total, 8 up, 8 in 2024-02-10T06:09:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[41707]: pgmap v803: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: osdmap e530: 8 total, 8 up, 8 in 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:43 smithi121 ceph-mon[30020]: pgmap v803: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:09:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[41707]: osdmap e531: 8 total, 8 up, 8 in 2024-02-10T06:09:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[30020]: osdmap e531: 8 total, 8 up, 8 in 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:45 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:45.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:09:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:45 smithi146 ceph-mon[33795]: osdmap e531: 8 total, 8 up, 8 in 2024-02-10T06:09:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:09:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:45 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:46.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:46 smithi146 ceph-mon[33795]: osdmap e532: 8 total, 8 up, 8 in 2024-02-10T06:09:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:46 smithi146 ceph-mon[33795]: pgmap v806: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:09:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:46 smithi146 ceph-mon[33795]: osdmap e533: 8 total, 8 up, 8 in 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[30020]: osdmap e532: 8 total, 8 up, 8 in 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[30020]: pgmap v806: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[30020]: osdmap e533: 8 total, 8 up, 8 in 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[41707]: osdmap e532: 8 total, 8 up, 8 in 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[41707]: pgmap v806: 228 pgs: 228 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi121-81810-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:09:46.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:46 smithi121 ceph-mon[41707]: osdmap e533: 8 total, 8 up, 8 in 2024-02-10T06:09:46.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:47 smithi146 ceph-mon[33795]: osdmap e534: 8 total, 8 up, 8 in 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[41707]: osdmap e534: 8 total, 8 up, 8 in 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:09:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:09:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:47 smithi121 ceph-mon[30020]: osdmap e534: 8 total, 8 up, 8 in 2024-02-10T06:09:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:48 smithi146 ceph-mon[33795]: pgmap v809: 236 pgs: 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:48 smithi146 ceph-mon[33795]: osdmap e535: 8 total, 8 up, 8 in 2024-02-10T06:09:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[41707]: pgmap v809: 236 pgs: 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[41707]: osdmap e535: 8 total, 8 up, 8 in 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[30020]: pgmap v809: 236 pgs: 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[30020]: osdmap e535: 8 total, 8 up, 8 in 2024-02-10T06:09:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:50.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:50.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:49 smithi146 ceph-mon[33795]: osdmap e536: 8 total, 8 up, 8 in 2024-02-10T06:09:50.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:49 smithi146 ceph-mon[33795]: pgmap v812: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:50.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:49 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[30020]: osdmap e536: 8 total, 8 up, 8 in 2024-02-10T06:09:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[30020]: pgmap v812: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:50.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[41707]: osdmap e536: 8 total, 8 up, 8 in 2024-02-10T06:09:50.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[41707]: pgmap v812: 268 pgs: 32 unknown, 8 creating+peering, 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:09:50.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:49 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:09:51.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:50 smithi146 ceph-mon[33795]: osdmap e537: 8 total, 8 up, 8 in 2024-02-10T06:09:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:51.229 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:50 smithi121 ceph-mon[41707]: osdmap e537: 8 total, 8 up, 8 in 2024-02-10T06:09:51.229 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:51.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:50 smithi121 ceph-mon[30020]: osdmap e537: 8 total, 8 up, 8 in 2024-02-10T06:09:51.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:09:52.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: osdmap e538: 8 total, 8 up, 8 in 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: pgmap v815: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: osdmap e539: 8 total, 8 up, 8 in 2024-02-10T06:09:52.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: osdmap e538: 8 total, 8 up, 8 in 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: pgmap v815: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: osdmap e539: 8 total, 8 up, 8 in 2024-02-10T06:09:52.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: osdmap e538: 8 total, 8 up, 8 in 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: pgmap v815: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: osdmap e539: 8 total, 8 up, 8 in 2024-02-10T06:09:52.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:53 smithi146 ceph-mon[33795]: osdmap e540: 8 total, 8 up, 8 in 2024-02-10T06:09:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:53 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:09:54.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:53 smithi146 ceph-mon[33795]: pgmap v818: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[41707]: osdmap e540: 8 total, 8 up, 8 in 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[41707]: pgmap v818: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-107", "overlaypool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[30020]: osdmap e540: 8 total, 8 up, 8 in 2024-02-10T06:09:54.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:09:54.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:53 smithi121 ceph-mon[30020]: pgmap v818: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:09:55.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:54 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]': finished 2024-02-10T06:09:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:54 smithi146 ceph-mon[33795]: osdmap e541: 8 total, 8 up, 8 in 2024-02-10T06:09:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:54 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]: dispatch 2024-02-10T06:09:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:09:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:09:55.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]': finished 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[30020]: osdmap e541: 8 total, 8 up, 8 in 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]: dispatch 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-107-cache", "mode": "writeback"}]': finished 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[41707]: osdmap e541: 8 total, 8 up, 8 in 2024-02-10T06:09:55.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:54 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]: dispatch 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]': finished 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: osdmap e542: 8 total, 8 up, 8 in 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: pgmap v821: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:56.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]': finished 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: osdmap e542: 8 total, 8 up, 8 in 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: pgmap v821: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:56.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:55 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:56.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-107"}]': finished 2024-02-10T06:09:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: osdmap e542: 8 total, 8 up, 8 in 2024-02-10T06:09:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]: dispatch 2024-02-10T06:09:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: pgmap v821: 300 pgs: 300 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:09:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:09:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:55 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1152201598' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-107", "tierpool": "test-rados-api-smithi121-81810-107-cache"}]': finished 2024-02-10T06:09:56.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:09:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[41707]: osdmap e543: 8 total, 8 up, 8 in 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[41707]: osdmap e544: 8 total, 8 up, 8 in 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[30020]: osdmap e543: 8 total, 8 up, 8 in 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:56 smithi121 ceph-mon[30020]: osdmap e544: 8 total, 8 up, 8 in 2024-02-10T06:09:57.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:56 smithi146 ceph-mon[33795]: osdmap e543: 8 total, 8 up, 8 in 2024-02-10T06:09:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:56 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:09:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:56 smithi146 ceph-mon[33795]: osdmap e544: 8 total, 8 up, 8 in 2024-02-10T06:09:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:57 smithi121 ceph-mon[41707]: pgmap v824: 268 pgs: 268 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:09:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:57 smithi121 ceph-mon[41707]: osdmap e545: 8 total, 8 up, 8 in 2024-02-10T06:09:58.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:57 smithi121 ceph-mon[30020]: pgmap v824: 268 pgs: 268 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:09:58.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:57 smithi121 ceph-mon[30020]: osdmap e545: 8 total, 8 up, 8 in 2024-02-10T06:09:58.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:57 smithi146 ceph-mon[33795]: pgmap v824: 268 pgs: 268 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:09:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:57 smithi146 ceph-mon[33795]: osdmap e545: 8 total, 8 up, 8 in 2024-02-10T06:09:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:59.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:09:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:09:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[41707]: osdmap e546: 8 total, 8 up, 8 in 2024-02-10T06:10:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[41707]: pgmap v827: 236 pgs: 236 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[30020]: osdmap e546: 8 total, 8 up, 8 in 2024-02-10T06:10:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:09:59 smithi121 ceph-mon[30020]: pgmap v827: 236 pgs: 236 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:00.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:59 smithi146 ceph-mon[33795]: osdmap e546: 8 total, 8 up, 8 in 2024-02-10T06:10:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:00.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:09:59 smithi146 ceph-mon[33795]: pgmap v827: 236 pgs: 236 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: osdmap e547: 8 total, 8 up, 8 in 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: application not enabled on pool 'LibRadosTierECPP_smithi121-81810-104' 2024-02-10T06:10:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[41707]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: osdmap e547: 8 total, 8 up, 8 in 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: application not enabled on pool 'LibRadosTierECPP_smithi121-81810-104' 2024-02-10T06:10:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:00 smithi121 ceph-mon[30020]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: osdmap e547: 8 total, 8 up, 8 in 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:10:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: application not enabled on pool 'LibRadosTierECPP_smithi121-81810-104' 2024-02-10T06:10:01.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:00 smithi146 ceph-mon[33795]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[30020]: osdmap e548: 8 total, 8 up, 8 in 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[30020]: pgmap v830: 244 pgs: 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:02.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[41707]: osdmap e548: 8 total, 8 up, 8 in 2024-02-10T06:10:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[41707]: pgmap v830: 244 pgs: 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:02.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:02.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:01 smithi146 ceph-mon[33795]: osdmap e548: 8 total, 8 up, 8 in 2024-02-10T06:10:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:01 smithi146 ceph-mon[33795]: pgmap v830: 244 pgs: 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:02.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[41707]: osdmap e549: 8 total, 8 up, 8 in 2024-02-10T06:10:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[30020]: osdmap e549: 8 total, 8 up, 8 in 2024-02-10T06:10:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:02 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:02 smithi146 ceph-mon[33795]: osdmap e549: 8 total, 8 up, 8 in 2024-02-10T06:10:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:02 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:10:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: osdmap e550: 8 total, 8 up, 8 in 2024-02-10T06:10:04.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: pgmap v833: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: osdmap e550: 8 total, 8 up, 8 in 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: pgmap v833: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:04.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:04.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:03 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: osdmap e550: 8 total, 8 up, 8 in 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: pgmap v833: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:10:04.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:03 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[41707]: osdmap e551: 8 total, 8 up, 8 in 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[30020]: osdmap e551: 8 total, 8 up, 8 in 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:05.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:04 smithi146 ceph-mon[33795]: osdmap e551: 8 total, 8 up, 8 in 2024-02-10T06:10:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:05.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:05 smithi146 ceph-mon[33795]: osdmap e552: 8 total, 8 up, 8 in 2024-02-10T06:10:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:05 smithi146 ceph-mon[33795]: pgmap v836: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[30020]: osdmap e552: 8 total, 8 up, 8 in 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[30020]: pgmap v836: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-109", "overlaypool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:06.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[41707]: osdmap e552: 8 total, 8 up, 8 in 2024-02-10T06:10:06.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]: dispatch 2024-02-10T06:10:06.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:05 smithi121 ceph-mon[41707]: pgmap v836: 276 pgs: 32 unknown, 8 creating+peering, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:06.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]': finished 2024-02-10T06:10:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: osdmap e553: 8 total, 8 up, 8 in 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]': finished 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: osdmap e553: 8 total, 8 up, 8 in 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:06 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:07.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-109-cache", "mode": "writeback"}]': finished 2024-02-10T06:10:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: osdmap e553: 8 total, 8 up, 8 in 2024-02-10T06:10:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:10:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:06 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: osdmap e554: 8 total, 8 up, 8 in 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: pgmap v839: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: osdmap e555: 8 total, 8 up, 8 in 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: osdmap e554: 8 total, 8 up, 8 in 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: pgmap v839: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: osdmap e555: 8 total, 8 up, 8 in 2024-02-10T06:10:08.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:08.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: osdmap e554: 8 total, 8 up, 8 in 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: pgmap v839: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: osdmap e555: 8 total, 8 up, 8 in 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:08.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:10:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:10:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[41707]: osdmap e556: 8 total, 8 up, 8 in 2024-02-10T06:10:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:10:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[30020]: osdmap e556: 8 total, 8 up, 8 in 2024-02-10T06:10:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:09.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:09 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:10:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:09 smithi146 ceph-mon[33795]: osdmap e556: 8 total, 8 up, 8 in 2024-02-10T06:10:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-10T06:10:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:10 smithi146 ceph-mon[33795]: pgmap v842: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T06:10:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:10 smithi146 ceph-mon[33795]: osdmap e557: 8 total, 8 up, 8 in 2024-02-10T06:10:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:10.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[30020]: pgmap v842: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T06:10:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[30020]: osdmap e557: 8 total, 8 up, 8 in 2024-02-10T06:10:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[41707]: pgmap v842: 276 pgs: 276 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-10T06:10:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[41707]: osdmap e557: 8 total, 8 up, 8 in 2024-02-10T06:10:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[41707]: osdmap e558: 8 total, 8 up, 8 in 2024-02-10T06:10:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[41707]: pgmap v845: 276 pgs: 276 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[30020]: osdmap e558: 8 total, 8 up, 8 in 2024-02-10T06:10:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:12 smithi121 ceph-mon[30020]: pgmap v845: 276 pgs: 276 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:12.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:12 smithi146 ceph-mon[33795]: osdmap e558: 8 total, 8 up, 8 in 2024-02-10T06:10:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]: dispatch 2024-02-10T06:10:12.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:12 smithi146 ceph-mon[33795]: pgmap v845: 276 pgs: 276 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:13 smithi121 ceph-mon[41707]: osdmap e559: 8 total, 8 up, 8 in 2024-02-10T06:10:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:13 smithi121 ceph-mon[30020]: osdmap e559: 8 total, 8 up, 8 in 2024-02-10T06:10:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-109", "tierpool": "test-rados-api-smithi121-81810-109-cache"}]': finished 2024-02-10T06:10:13.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:13 smithi146 ceph-mon[33795]: osdmap e559: 8 total, 8 up, 8 in 2024-02-10T06:10:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:14 smithi121 ceph-mon[41707]: osdmap e560: 8 total, 8 up, 8 in 2024-02-10T06:10:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:14 smithi121 ceph-mon[41707]: pgmap v848: 244 pgs: 244 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:14 smithi121 ceph-mon[30020]: osdmap e560: 8 total, 8 up, 8 in 2024-02-10T06:10:14.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:14 smithi121 ceph-mon[30020]: pgmap v848: 244 pgs: 244 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:14.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:14 smithi146 ceph-mon[33795]: osdmap e560: 8 total, 8 up, 8 in 2024-02-10T06:10:14.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:14 smithi146 ceph-mon[33795]: pgmap v848: 244 pgs: 244 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[41707]: osdmap e561: 8 total, 8 up, 8 in 2024-02-10T06:10:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[30020]: osdmap e561: 8 total, 8 up, 8 in 2024-02-10T06:10:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:15 smithi146 ceph-mon[33795]: osdmap e561: 8 total, 8 up, 8 in 2024-02-10T06:10:15.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[41707]: osdmap e562: 8 total, 8 up, 8 in 2024-02-10T06:10:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[41707]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[30020]: osdmap e562: 8 total, 8 up, 8 in 2024-02-10T06:10:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:16 smithi121 ceph-mon[30020]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:16.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:16.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-mon[33795]: osdmap e562: 8 total, 8 up, 8 in 2024-02-10T06:10:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2849663459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]: dispatch 2024-02-10T06:10:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-mon[33795]: pgmap v851: 236 pgs: 236 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:16.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[41707]: osdmap e563: 8 total, 8 up, 8 in 2024-02-10T06:10:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[30020]: osdmap e563: 8 total, 8 up, 8 in 2024-02-10T06:10:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:17.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:17 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-109"}]': finished 2024-02-10T06:10:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:17 smithi146 ceph-mon[33795]: osdmap e563: 8 total, 8 up, 8 in 2024-02-10T06:10:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[30020]: osdmap e564: 8 total, 8 up, 8 in 2024-02-10T06:10:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[30020]: pgmap v854: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:18.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[41707]: osdmap e564: 8 total, 8 up, 8 in 2024-02-10T06:10:18.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:18 smithi121 ceph-mon[41707]: pgmap v854: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:18 smithi146 ceph-mon[33795]: osdmap e564: 8 total, 8 up, 8 in 2024-02-10T06:10:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:18 smithi146 ceph-mon[33795]: pgmap v854: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[41707]: osdmap e565: 8 total, 8 up, 8 in 2024-02-10T06:10:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[30020]: osdmap e565: 8 total, 8 up, 8 in 2024-02-10T06:10:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:19 smithi146 ceph-mon[33795]: osdmap e565: 8 total, 8 up, 8 in 2024-02-10T06:10:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1753417239' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:19.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: osdmap e566: 8 total, 8 up, 8 in 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:20 smithi146 ceph-mon[33795]: pgmap v857: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: osdmap e566: 8 total, 8 up, 8 in 2024-02-10T06:10:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[41707]: pgmap v857: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi121-81810-104"}]': finished 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: osdmap e566: 8 total, 8 up, 8 in 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-10T06:10:20.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:20 smithi121 ceph-mon[30020]: pgmap v857: 228 pgs: 228 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:10:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[41707]: osdmap e567: 8 total, 8 up, 8 in 2024-02-10T06:10:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:21.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:21.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:21.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[30020]: osdmap e567: 8 total, 8 up, 8 in 2024-02-10T06:10:21.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:21.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:21.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi121-81810-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-10T06:10:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:21 smithi146 ceph-mon[33795]: osdmap e567: 8 total, 8 up, 8 in 2024-02-10T06:10:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:21.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:22 smithi146 ceph-mon[33795]: pgmap v860: 228 pgs: 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:22 smithi146 ceph-mon[33795]: osdmap e568: 8 total, 8 up, 8 in 2024-02-10T06:10:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:22.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:22 smithi146 ceph-mon[33795]: osdmap e569: 8 total, 8 up, 8 in 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[41707]: pgmap v860: 228 pgs: 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[41707]: osdmap e568: 8 total, 8 up, 8 in 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[41707]: osdmap e569: 8 total, 8 up, 8 in 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[30020]: pgmap v860: 228 pgs: 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[30020]: osdmap e568: 8 total, 8 up, 8 in 2024-02-10T06:10:22.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi121-81810-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:22.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:22 smithi121 ceph-mon[30020]: osdmap e569: 8 total, 8 up, 8 in 2024-02-10T06:10:24.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:24 smithi146 ceph-mon[33795]: osdmap e570: 8 total, 8 up, 8 in 2024-02-10T06:10:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:24 smithi146 ceph-mon[33795]: pgmap v863: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[41707]: osdmap e570: 8 total, 8 up, 8 in 2024-02-10T06:10:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[41707]: pgmap v863: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:24.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[30020]: osdmap e570: 8 total, 8 up, 8 in 2024-02-10T06:10:24.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:24.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:24 smithi121 ceph-mon[30020]: pgmap v863: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:25.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[30020]: osdmap e571: 8 total, 8 up, 8 in 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[41707]: osdmap e571: 8 total, 8 up, 8 in 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:25.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:25 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:25 smithi146 ceph-mon[33795]: osdmap e571: 8 total, 8 up, 8 in 2024-02-10T06:10:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-mon[33795]: osdmap e572: 8 total, 8 up, 8 in 2024-02-10T06:10:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-mon[33795]: pgmap v866: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[41707]: osdmap e572: 8 total, 8 up, 8 in 2024-02-10T06:10:26.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[41707]: pgmap v866: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[30020]: osdmap e572: 8 total, 8 up, 8 in 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:26.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:26 smithi121 ceph-mon[30020]: pgmap v866: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:26.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:27 smithi146 ceph-mon[33795]: osdmap e573: 8 total, 8 up, 8 in 2024-02-10T06:10:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:27 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[30020]: osdmap e573: 8 total, 8 up, 8 in 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:27.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:27.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[41707]: osdmap e573: 8 total, 8 up, 8 in 2024-02-10T06:10:27.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:27.580 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:28 smithi146 ceph-mon[33795]: osdmap e574: 8 total, 8 up, 8 in 2024-02-10T06:10:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:28 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:28 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:28 smithi146 ceph-mon[33795]: pgmap v869: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[41707]: osdmap e574: 8 total, 8 up, 8 in 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[41707]: pgmap v869: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[30020]: osdmap e574: 8 total, 8 up, 8 in 2024-02-10T06:10:28.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]: dispatch 2024-02-10T06:10:28.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:28 smithi121 ceph-mon[30020]: pgmap v869: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:10:29.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:29 smithi146 ceph-mon[33795]: osdmap e575: 8 total, 8 up, 8 in 2024-02-10T06:10:29.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:29.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:29 smithi121 ceph-mon[30020]: osdmap e575: 8 total, 8 up, 8 in 2024-02-10T06:10:29.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-112"}]': finished 2024-02-10T06:10:29.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:29 smithi121 ceph-mon[41707]: osdmap e575: 8 total, 8 up, 8 in 2024-02-10T06:10:30.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:30 smithi146 ceph-mon[33795]: osdmap e576: 8 total, 8 up, 8 in 2024-02-10T06:10:30.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:30 smithi146 ceph-mon[33795]: pgmap v872: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:30.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:30 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[41707]: osdmap e576: 8 total, 8 up, 8 in 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[41707]: pgmap v872: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[30020]: osdmap e576: 8 total, 8 up, 8 in 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[30020]: pgmap v872: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:30.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:30 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:31.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:31 smithi146 ceph-mon[33795]: osdmap e577: 8 total, 8 up, 8 in 2024-02-10T06:10:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:31.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[30020]: osdmap e577: 8 total, 8 up, 8 in 2024-02-10T06:10:31.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:31.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:31.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[41707]: osdmap e577: 8 total, 8 up, 8 in 2024-02-10T06:10:31.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:31.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:32.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: osdmap e578: 8 total, 8 up, 8 in 2024-02-10T06:10:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: pgmap v875: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:32.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:32.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: osdmap e578: 8 total, 8 up, 8 in 2024-02-10T06:10:32.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: pgmap v875: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:32.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: osdmap e578: 8 total, 8 up, 8 in 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: pgmap v875: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:32.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:33 smithi146 ceph-mon[33795]: osdmap e579: 8 total, 8 up, 8 in 2024-02-10T06:10:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[41707]: osdmap e579: 8 total, 8 up, 8 in 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:33.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[30020]: osdmap e579: 8 total, 8 up, 8 in 2024-02-10T06:10:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:33.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: osdmap e580: 8 total, 8 up, 8 in 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: pgmap v878: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]': finished 2024-02-10T06:10:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:34 smithi146 ceph-mon[33795]: osdmap e581: 8 total, 8 up, 8 in 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: osdmap e580: 8 total, 8 up, 8 in 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: pgmap v878: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:34.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]': finished 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[41707]: osdmap e581: 8 total, 8 up, 8 in 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: osdmap e580: 8 total, 8 up, 8 in 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]: dispatch 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: pgmap v878: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T06:10:34.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:34.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-114", "mode": "writeback"}]': finished 2024-02-10T06:10:34.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:34 smithi121 ceph-mon[30020]: osdmap e581: 8 total, 8 up, 8 in 2024-02-10T06:10:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:35.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:35.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:35.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:10:36.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-mon[33795]: pgmap v880: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:36.568 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:36.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-mon[33795]: osdmap e582: 8 total, 8 up, 8 in 2024-02-10T06:10:36.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.569 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[41707]: pgmap v880: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[41707]: osdmap e582: 8 total, 8 up, 8 in 2024-02-10T06:10:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[30020]: pgmap v880: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:10:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:10:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[30020]: osdmap e582: 8 total, 8 up, 8 in 2024-02-10T06:10:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]: dispatch 2024-02-10T06:10:36.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[41707]: osdmap e583: 8 total, 8 up, 8 in 2024-02-10T06:10:37.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:37.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[30020]: osdmap e583: 8 total, 8 up, 8 in 2024-02-10T06:10:37.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:37.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:37 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:10:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-114"}]': finished 2024-02-10T06:10:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:37 smithi146 ceph-mon[33795]: osdmap e583: 8 total, 8 up, 8 in 2024-02-10T06:10:37.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[41707]: pgmap v883: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[41707]: osdmap e584: 8 total, 8 up, 8 in 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[30020]: pgmap v883: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:38.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:38 smithi121 ceph-mon[30020]: osdmap e584: 8 total, 8 up, 8 in 2024-02-10T06:10:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:38 smithi146 ceph-mon[33795]: pgmap v883: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:10:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:38 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:38.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:38 smithi146 ceph-mon[33795]: osdmap e584: 8 total, 8 up, 8 in 2024-02-10T06:10:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[41707]: osdmap e585: 8 total, 8 up, 8 in 2024-02-10T06:10:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:39.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:39.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[30020]: osdmap e585: 8 total, 8 up, 8 in 2024-02-10T06:10:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:39.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:39 smithi146 ceph-mon[33795]: osdmap e585: 8 total, 8 up, 8 in 2024-02-10T06:10:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:39.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:10:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[41707]: pgmap v886: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:40.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[41707]: osdmap e586: 8 total, 8 up, 8 in 2024-02-10T06:10:40.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[30020]: pgmap v886: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:40.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:40 smithi121 ceph-mon[30020]: osdmap e586: 8 total, 8 up, 8 in 2024-02-10T06:10:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:40 smithi146 ceph-mon[33795]: pgmap v886: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:10:40.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:10:40.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:40 smithi146 ceph-mon[33795]: osdmap e586: 8 total, 8 up, 8 in 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[41707]: osdmap e587: 8 total, 8 up, 8 in 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[30020]: osdmap e587: 8 total, 8 up, 8 in 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:41.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:41.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:41 smithi146 ceph-mon[33795]: osdmap e587: 8 total, 8 up, 8 in 2024-02-10T06:10:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:41.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[41707]: pgmap v889: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[41707]: osdmap e588: 8 total, 8 up, 8 in 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[30020]: pgmap v889: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:10:42.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:42.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[30020]: osdmap e588: 8 total, 8 up, 8 in 2024-02-10T06:10:42.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:42.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:42.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:42 smithi146 ceph-mon[33795]: pgmap v889: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-10T06:10:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:10:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:42 smithi146 ceph-mon[33795]: osdmap e588: 8 total, 8 up, 8 in 2024-02-10T06:10:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[41707]: osdmap e589: 8 total, 8 up, 8 in 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:10:43.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[30020]: osdmap e589: 8 total, 8 up, 8 in 2024-02-10T06:10:43.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:43.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:10:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:43 smithi146 ceph-mon[33795]: osdmap e589: 8 total, 8 up, 8 in 2024-02-10T06:10:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]: dispatch 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]': finished 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: osdmap e590: 8 total, 8 up, 8 in 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: 315.7 scrub starts 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[30020]: 315.7 scrub ok 2024-02-10T06:10:44.241 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]': finished 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: osdmap e590: 8 total, 8 up, 8 in 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: 315.7 scrub starts 2024-02-10T06:10:44.242 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:44 smithi121 ceph-mon[41707]: 315.7 scrub ok 2024-02-10T06:10:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:10:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:10:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-116", "mode": "writeback"}]': finished 2024-02-10T06:10:44.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: osdmap e590: 8 total, 8 up, 8 in 2024-02-10T06:10:44.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.7"}]: dispatch 2024-02-10T06:10:44.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: 315.7 scrub starts 2024-02-10T06:10:44.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:44 smithi146 ceph-mon[33795]: 315.7 scrub ok 2024-02-10T06:10:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:45 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:45 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:45.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:45 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:10:46.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:46 smithi146 ceph-mon[33795]: pgmap v894: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-10T06:10:46.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:46 smithi121 ceph-mon[41707]: pgmap v894: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-10T06:10:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:46 smithi121 ceph-mon[30020]: pgmap v894: 268 pgs: 268 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-02-10T06:10:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:47 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:10:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:10:48.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:48 smithi146 ceph-mon[33795]: pgmap v895: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:48 smithi121 ceph-mon[41707]: pgmap v895: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:48 smithi121 ceph-mon[30020]: pgmap v895: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:50 smithi146 ceph-mon[33795]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 1 op/s 2024-02-10T06:10:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:50 smithi121 ceph-mon[41707]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 1 op/s 2024-02-10T06:10:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:50 smithi121 ceph-mon[30020]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 1 op/s 2024-02-10T06:10:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:52 smithi146 ceph-mon[33795]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:52 smithi121 ceph-mon[30020]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:52 smithi121 ceph-mon[41707]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:54 smithi146 ceph-mon[33795]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:54 smithi121 ceph-mon[41707]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:54 smithi121 ceph-mon[30020]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:10:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:10:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:10:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:10:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:56 smithi146 ceph-mon[33795]: pgmap v899: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:10:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:10:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:10:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:56 smithi121 ceph-mon[30020]: pgmap v899: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:10:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:56 smithi121 ceph-mon[41707]: pgmap v899: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:10:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:10:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:58 smithi121 ceph-mon[30020]: pgmap v900: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:58 smithi121 ceph-mon[41707]: pgmap v900: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:10:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:58 smithi146 ceph-mon[33795]: pgmap v900: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:11:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[30020]: osdmap e591: 8 total, 8 up, 8 in 2024-02-10T06:11:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:00.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[41707]: osdmap e591: 8 total, 8 up, 8 in 2024-02-10T06:11:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:00.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:10:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:00.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:59 smithi146 ceph-mon[33795]: osdmap e591: 8 total, 8 up, 8 in 2024-02-10T06:11:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:10:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[30020]: pgmap v902: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[30020]: osdmap e592: 8 total, 8 up, 8 in 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[41707]: pgmap v902: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:11:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[41707]: osdmap e592: 8 total, 8 up, 8 in 2024-02-10T06:11:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:00 smithi146 ceph-mon[33795]: pgmap v902: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T06:11:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:00 smithi146 ceph-mon[33795]: osdmap e592: 8 total, 8 up, 8 in 2024-02-10T06:11:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]: dispatch 2024-02-10T06:11:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:11:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[30020]: osdmap e593: 8 total, 8 up, 8 in 2024-02-10T06:11:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:02.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:11:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[41707]: osdmap e593: 8 total, 8 up, 8 in 2024-02-10T06:11:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:01 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:02.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:01 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-116"}]': finished 2024-02-10T06:11:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:01 smithi146 ceph-mon[33795]: osdmap e593: 8 total, 8 up, 8 in 2024-02-10T06:11:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:01 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:01 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[30020]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:11:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[30020]: osdmap e594: 8 total, 8 up, 8 in 2024-02-10T06:11:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[41707]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:11:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:02 smithi121 ceph-mon[41707]: osdmap e594: 8 total, 8 up, 8 in 2024-02-10T06:11:03.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:02 smithi146 ceph-mon[33795]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:11:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:02 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:02 smithi146 ceph-mon[33795]: osdmap e594: 8 total, 8 up, 8 in 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[30020]: osdmap e595: 8 total, 8 up, 8 in 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[41707]: osdmap e595: 8 total, 8 up, 8 in 2024-02-10T06:11:04.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:04.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:03 smithi146 ceph-mon[33795]: osdmap e595: 8 total, 8 up, 8 in 2024-02-10T06:11:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:04.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:11:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: pgmap v908: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: osdmap e596: 8 total, 8 up, 8 in 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: osdmap e597: 8 total, 8 up, 8 in 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: pgmap v908: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: osdmap e596: 8 total, 8 up, 8 in 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: osdmap e597: 8 total, 8 up, 8 in 2024-02-10T06:11:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.012 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: pgmap v908: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: osdmap e596: 8 total, 8 up, 8 in 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: osdmap e597: 8 total, 8 up, 8 in 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:05.273 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-mon[33795]: pgmap v911: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-mon[33795]: osdmap e598: 8 total, 8 up, 8 in 2024-02-10T06:11:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:06.673 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[41707]: pgmap v911: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[41707]: osdmap e598: 8 total, 8 up, 8 in 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[30020]: pgmap v911: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[30020]: osdmap e598: 8 total, 8 up, 8 in 2024-02-10T06:11:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:07.658 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: osdmap e599: 8 total, 8 up, 8 in 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.659 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: osdmap e599: 8 total, 8 up, 8 in 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: osdmap e599: 8 total, 8 up, 8 in 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]: dispatch 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:08.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]': finished 2024-02-10T06:11:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: osdmap e600: 8 total, 8 up, 8 in 2024-02-10T06:11:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:08.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:08 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]': finished 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: osdmap e600: 8 total, 8 up, 8 in 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:08.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-118", "mode": "writeback"}]': finished 2024-02-10T06:11:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: osdmap e600: 8 total, 8 up, 8 in 2024-02-10T06:11:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:08 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: osdmap e601: 8 total, 8 up, 8 in 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:09.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: osdmap e601: 8 total, 8 up, 8 in 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:09 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: osdmap e601: 8 total, 8 up, 8 in 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:09 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: osdmap e602: 8 total, 8 up, 8 in 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: osdmap e602: 8 total, 8 up, 8 in 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:10.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: osdmap e602: 8 total, 8 up, 8 in 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]: dispatch 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-118"}]': finished 2024-02-10T06:11:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:11 smithi121 ceph-mon[41707]: osdmap e603: 8 total, 8 up, 8 in 2024-02-10T06:11:11.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:11 smithi121 ceph-mon[30020]: osdmap e603: 8 total, 8 up, 8 in 2024-02-10T06:11:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:11 smithi146 ceph-mon[33795]: osdmap e603: 8 total, 8 up, 8 in 2024-02-10T06:11:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:12 smithi121 ceph-mon[41707]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:11:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:12 smithi121 ceph-mon[41707]: osdmap e604: 8 total, 8 up, 8 in 2024-02-10T06:11:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:12 smithi121 ceph-mon[30020]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:11:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:12 smithi121 ceph-mon[30020]: osdmap e604: 8 total, 8 up, 8 in 2024-02-10T06:11:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:12 smithi146 ceph-mon[33795]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:11:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:12 smithi146 ceph-mon[33795]: osdmap e604: 8 total, 8 up, 8 in 2024-02-10T06:11:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[41707]: osdmap e605: 8 total, 8 up, 8 in 2024-02-10T06:11:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[30020]: osdmap e605: 8 total, 8 up, 8 in 2024-02-10T06:11:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:13 smithi146 ceph-mon[33795]: osdmap e605: 8 total, 8 up, 8 in 2024-02-10T06:11:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: pgmap v923: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: osdmap e606: 8 total, 8 up, 8 in 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: osdmap e607: 8 total, 8 up, 8 in 2024-02-10T06:11:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: pgmap v923: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: osdmap e606: 8 total, 8 up, 8 in 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: osdmap e607: 8 total, 8 up, 8 in 2024-02-10T06:11:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:14.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: pgmap v923: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: osdmap e606: 8 total, 8 up, 8 in 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: osdmap e607: 8 total, 8 up, 8 in 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:16.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-mon[33795]: pgmap v926: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-mon[33795]: osdmap e608: 8 total, 8 up, 8 in 2024-02-10T06:11:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:16.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[41707]: pgmap v926: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[41707]: osdmap e608: 8 total, 8 up, 8 in 2024-02-10T06:11:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[30020]: pgmap v926: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[30020]: osdmap e608: 8 total, 8 up, 8 in 2024-02-10T06:11:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:16.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]: dispatch 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]': finished 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: osdmap e609: 8 total, 8 up, 8 in 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:17.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]': finished 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: osdmap e609: 8 total, 8 up, 8 in 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-120", "mode": "writeback"}]': finished 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: osdmap e609: 8 total, 8 up, 8 in 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:17.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:17.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:18 smithi146 ceph-mon[33795]: pgmap v929: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:18 smithi146 ceph-mon[33795]: osdmap e610: 8 total, 8 up, 8 in 2024-02-10T06:11:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:18.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[41707]: pgmap v929: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[41707]: osdmap e610: 8 total, 8 up, 8 in 2024-02-10T06:11:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[30020]: pgmap v929: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[30020]: osdmap e610: 8 total, 8 up, 8 in 2024-02-10T06:11:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]: dispatch 2024-02-10T06:11:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[41707]: osdmap e611: 8 total, 8 up, 8 in 2024-02-10T06:11:19.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:19.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[30020]: osdmap e611: 8 total, 8 up, 8 in 2024-02-10T06:11:19.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:19 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:19.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:19 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-120"}]': finished 2024-02-10T06:11:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:19 smithi146 ceph-mon[33795]: osdmap e611: 8 total, 8 up, 8 in 2024-02-10T06:11:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:19 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:20 smithi121 ceph-mon[41707]: pgmap v932: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:20 smithi121 ceph-mon[41707]: osdmap e612: 8 total, 8 up, 8 in 2024-02-10T06:11:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:20 smithi121 ceph-mon[30020]: pgmap v932: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:20 smithi121 ceph-mon[30020]: osdmap e612: 8 total, 8 up, 8 in 2024-02-10T06:11:20.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:20 smithi146 ceph-mon[33795]: pgmap v932: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:20 smithi146 ceph-mon[33795]: osdmap e612: 8 total, 8 up, 8 in 2024-02-10T06:11:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[41707]: osdmap e613: 8 total, 8 up, 8 in 2024-02-10T06:11:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:21.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[30020]: osdmap e613: 8 total, 8 up, 8 in 2024-02-10T06:11:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:21.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:21.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:21 smithi146 ceph-mon[33795]: osdmap e613: 8 total, 8 up, 8 in 2024-02-10T06:11:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:22.729 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[41707]: pgmap v935: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[41707]: osdmap e614: 8 total, 8 up, 8 in 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[30020]: pgmap v935: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[30020]: osdmap e614: 8 total, 8 up, 8 in 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:22.730 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:22 smithi146 ceph-mon[33795]: pgmap v935: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:22 smithi146 ceph-mon[33795]: osdmap e614: 8 total, 8 up, 8 in 2024-02-10T06:11:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:23.745 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[41707]: osdmap e615: 8 total, 8 up, 8 in 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[30020]: osdmap e615: 8 total, 8 up, 8 in 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:23.746 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:23.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:23 smithi146 ceph-mon[33795]: osdmap e615: 8 total, 8 up, 8 in 2024-02-10T06:11:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:23.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[30020]: pgmap v938: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[30020]: osdmap e616: 8 total, 8 up, 8 in 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[41707]: pgmap v938: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[41707]: osdmap e616: 8 total, 8 up, 8 in 2024-02-10T06:11:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:24.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:24.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:24 smithi146 ceph-mon[33795]: pgmap v938: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:24 smithi146 ceph-mon[33795]: osdmap e616: 8 total, 8 up, 8 in 2024-02-10T06:11:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]: dispatch 2024-02-10T06:11:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]': finished 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[41707]: osdmap e617: 8 total, 8 up, 8 in 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:25.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]': finished 2024-02-10T06:11:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[30020]: osdmap e617: 8 total, 8 up, 8 in 2024-02-10T06:11:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:25.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:25.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:25 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-122", "mode": "writeback"}]': finished 2024-02-10T06:11:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:25 smithi146 ceph-mon[33795]: osdmap e617: 8 total, 8 up, 8 in 2024-02-10T06:11:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:25.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[41707]: pgmap v941: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[41707]: osdmap e618: 8 total, 8 up, 8 in 2024-02-10T06:11:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[30020]: pgmap v941: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[30020]: osdmap e618: 8 total, 8 up, 8 in 2024-02-10T06:11:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:26.921 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-mon[33795]: pgmap v941: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-mon[33795]: osdmap e618: 8 total, 8 up, 8 in 2024-02-10T06:11:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]: dispatch 2024-02-10T06:11:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[41707]: osdmap e619: 8 total, 8 up, 8 in 2024-02-10T06:11:27.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[30020]: osdmap e619: 8 total, 8 up, 8 in 2024-02-10T06:11:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:27.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:27 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-122"}]': finished 2024-02-10T06:11:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:27 smithi146 ceph-mon[33795]: osdmap e619: 8 total, 8 up, 8 in 2024-02-10T06:11:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[41707]: pgmap v944: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T06:11:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[41707]: osdmap e620: 8 total, 8 up, 8 in 2024-02-10T06:11:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[30020]: pgmap v944: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T06:11:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:28 smithi121 ceph-mon[30020]: osdmap e620: 8 total, 8 up, 8 in 2024-02-10T06:11:28.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:28 smithi146 ceph-mon[33795]: pgmap v944: 268 pgs: 268 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-10T06:11:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:28 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:28 smithi146 ceph-mon[33795]: osdmap e620: 8 total, 8 up, 8 in 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[30020]: osdmap e621: 8 total, 8 up, 8 in 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[41707]: osdmap e621: 8 total, 8 up, 8 in 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:29 smithi146 ceph-mon[33795]: osdmap e621: 8 total, 8 up, 8 in 2024-02-10T06:11:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[30020]: pgmap v947: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:11:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[30020]: osdmap e622: 8 total, 8 up, 8 in 2024-02-10T06:11:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[41707]: pgmap v947: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:11:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:30.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:30 smithi121 ceph-mon[41707]: osdmap e622: 8 total, 8 up, 8 in 2024-02-10T06:11:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:30 smithi146 ceph-mon[33795]: pgmap v947: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-10T06:11:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:30 smithi146 ceph-mon[33795]: osdmap e622: 8 total, 8 up, 8 in 2024-02-10T06:11:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[41707]: osdmap e623: 8 total, 8 up, 8 in 2024-02-10T06:11:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[30020]: osdmap e623: 8 total, 8 up, 8 in 2024-02-10T06:11:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:31 smithi146 ceph-mon[33795]: osdmap e623: 8 total, 8 up, 8 in 2024-02-10T06:11:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: pgmap v950: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-02-10T06:11:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: osdmap e624: 8 total, 8 up, 8 in 2024-02-10T06:11:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: pgmap v950: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: osdmap e624: 8 total, 8 up, 8 in 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:32.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: pgmap v950: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 1 op/s 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: osdmap e624: 8 total, 8 up, 8 in 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:11:32.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[41707]: osdmap e625: 8 total, 8 up, 8 in 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[30020]: osdmap e625: 8 total, 8 up, 8 in 2024-02-10T06:11:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:33.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:33.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:33 smithi146 ceph-mon[33795]: osdmap e625: 8 total, 8 up, 8 in 2024-02-10T06:11:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]: dispatch 2024-02-10T06:11:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[41707]: pgmap v953: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:11:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]': finished 2024-02-10T06:11:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[41707]: osdmap e626: 8 total, 8 up, 8 in 2024-02-10T06:11:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[30020]: pgmap v953: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:11:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]': finished 2024-02-10T06:11:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[30020]: osdmap e626: 8 total, 8 up, 8 in 2024-02-10T06:11:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:34 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:34.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:34 smithi146 ceph-mon[33795]: pgmap v953: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-10T06:11:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:34 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-124", "mode": "writeback"}]': finished 2024-02-10T06:11:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:34 smithi146 ceph-mon[33795]: osdmap e626: 8 total, 8 up, 8 in 2024-02-10T06:11:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:34 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[41707]: osdmap e627: 8 total, 8 up, 8 in 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[30020]: osdmap e627: 8 total, 8 up, 8 in 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:35.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:35 smithi146 ceph-mon[33795]: osdmap e627: 8 total, 8 up, 8 in 2024-02-10T06:11:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:35 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:36.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:36.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-mon[33795]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:36.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:36.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-mon[33795]: osdmap e628: 8 total, 8 up, 8 in 2024-02-10T06:11:36.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:36.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[41707]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[41707]: osdmap e628: 8 total, 8 up, 8 in 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[30020]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[30020]: osdmap e628: 8 total, 8 up, 8 in 2024-02-10T06:11:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]: dispatch 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[30020]: osdmap e629: 8 total, 8 up, 8 in 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:37.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[41707]: osdmap e629: 8 total, 8 up, 8 in 2024-02-10T06:11:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:37.843 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:37 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:37.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-124"}]': finished 2024-02-10T06:11:37.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:37 smithi146 ceph-mon[33795]: osdmap e629: 8 total, 8 up, 8 in 2024-02-10T06:11:37.844 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:38 smithi121 ceph-mon[41707]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:11:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:38 smithi121 ceph-mon[41707]: osdmap e630: 8 total, 8 up, 8 in 2024-02-10T06:11:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:38 smithi121 ceph-mon[30020]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:11:38.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:38 smithi121 ceph-mon[30020]: osdmap e630: 8 total, 8 up, 8 in 2024-02-10T06:11:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:38 smithi146 ceph-mon[33795]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s 2024-02-10T06:11:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:38 smithi146 ceph-mon[33795]: osdmap e630: 8 total, 8 up, 8 in 2024-02-10T06:11:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: osdmap e631: 8 total, 8 up, 8 in 2024-02-10T06:11:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: osdmap e632: 8 total, 8 up, 8 in 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: osdmap e631: 8 total, 8 up, 8 in 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: osdmap e632: 8 total, 8 up, 8 in 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:39.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: osdmap e631: 8 total, 8 up, 8 in 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: osdmap e632: 8 total, 8 up, 8 in 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[41707]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[41707]: osdmap e633: 8 total, 8 up, 8 in 2024-02-10T06:11:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[30020]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[30020]: osdmap e633: 8 total, 8 up, 8 in 2024-02-10T06:11:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:40 smithi146 ceph-mon[33795]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:40 smithi146 ceph-mon[33795]: osdmap e633: 8 total, 8 up, 8 in 2024-02-10T06:11:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:42 smithi146 ceph-mon[33795]: pgmap v965: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:42 smithi146 ceph-mon[33795]: osdmap e634: 8 total, 8 up, 8 in 2024-02-10T06:11:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:42.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[41707]: pgmap v965: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[41707]: osdmap e634: 8 total, 8 up, 8 in 2024-02-10T06:11:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[30020]: pgmap v965: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[30020]: osdmap e634: 8 total, 8 up, 8 in 2024-02-10T06:11:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]: dispatch 2024-02-10T06:11:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:43 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]': finished 2024-02-10T06:11:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:43 smithi146 ceph-mon[33795]: osdmap e635: 8 total, 8 up, 8 in 2024-02-10T06:11:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:43.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]': finished 2024-02-10T06:11:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[41707]: osdmap e635: 8 total, 8 up, 8 in 2024-02-10T06:11:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-126", "mode": "writeback"}]': finished 2024-02-10T06:11:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[30020]: osdmap e635: 8 total, 8 up, 8 in 2024-02-10T06:11:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: pgmap v968: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: osdmap e636: 8 total, 8 up, 8 in 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: pgmap v968: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: osdmap e636: 8 total, 8 up, 8 in 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:44 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: pgmap v968: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: osdmap e636: 8 total, 8 up, 8 in 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]: dispatch 2024-02-10T06:11:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:44 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:45.442 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.442 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9037 ms) 2024-02-10T06:11:45.442 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-10T06:11:45.442 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.442 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6045 ms) 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5041 ms) 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25471 ms) 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13297 ms) 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12088 ms) 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-10T06:11:45.443 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 00000012 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12064 ms) 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: require_osd_release = squid 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6031 ms) 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (555310 ms total) 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1015 ms) 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-10T06:11:45.444 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11025 ms) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18138 ms) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (1 ms) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30179 ms total) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7042 ms) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8075 ms) 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: waiting for scrub... 2024-02-10T06:11:45.445 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: done waiting 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24503 ms) 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9731 ms) 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7994 ms) 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8054 ms) 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10076 ms) 2024-02-10T06:11:45.446 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-10T06:11:45.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:45.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:45.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[41707]: osdmap e637: 8 total, 8 up, 8 in 2024-02-10T06:11:45.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:45.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:45.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:45 smithi121 ceph-mon[30020]: osdmap e637: 8 total, 8 up, 8 in 2024-02-10T06:11:45.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:45 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:45.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-126"}]': finished 2024-02-10T06:11:45.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:45 smithi146 ceph-mon[33795]: osdmap e637: 8 total, 8 up, 8 in 2024-02-10T06:11:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:46 smithi121 ceph-mon[41707]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:46 smithi121 ceph-mon[41707]: osdmap e638: 8 total, 8 up, 8 in 2024-02-10T06:11:46.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:46 smithi121 ceph-mon[30020]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:46 smithi121 ceph-mon[30020]: osdmap e638: 8 total, 8 up, 8 in 2024-02-10T06:11:46.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:46 smithi146 ceph-mon[33795]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:11:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:46 smithi146 ceph-mon[33795]: osdmap e638: 8 total, 8 up, 8 in 2024-02-10T06:11:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[30020]: osdmap e639: 8 total, 8 up, 8 in 2024-02-10T06:11:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[41707]: osdmap e639: 8 total, 8 up, 8 in 2024-02-10T06:11:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:47 smithi146 ceph-mon[33795]: osdmap e639: 8 total, 8 up, 8 in 2024-02-10T06:11:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:11:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[30020]: pgmap v974: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[30020]: osdmap e640: 8 total, 8 up, 8 in 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[41707]: pgmap v974: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[41707]: osdmap e640: 8 total, 8 up, 8 in 2024-02-10T06:11:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:48.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:48 smithi146 ceph-mon[33795]: pgmap v974: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:11:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:48 smithi146 ceph-mon[33795]: osdmap e640: 8 total, 8 up, 8 in 2024-02-10T06:11:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:11:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[41707]: osdmap e641: 8 total, 8 up, 8 in 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[30020]: osdmap e641: 8 total, 8 up, 8 in 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:49 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:11:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:49 smithi146 ceph-mon[33795]: osdmap e641: 8 total, 8 up, 8 in 2024-02-10T06:11:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:49 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[41707]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[41707]: osdmap e642: 8 total, 8 up, 8 in 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[30020]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[30020]: osdmap e642: 8 total, 8 up, 8 in 2024-02-10T06:11:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:50 smithi146 ceph-mon[33795]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:11:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:50 smithi146 ceph-mon[33795]: osdmap e642: 8 total, 8 up, 8 in 2024-02-10T06:11:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]: dispatch 2024-02-10T06:11:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]': finished 2024-02-10T06:11:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[41707]: osdmap e643: 8 total, 8 up, 8 in 2024-02-10T06:11:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]': finished 2024-02-10T06:11:51.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:51 smithi121 ceph-mon[30020]: osdmap e643: 8 total, 8 up, 8 in 2024-02-10T06:11:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:51 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:11:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-128", "mode": "writeback"}]': finished 2024-02-10T06:11:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:51 smithi146 ceph-mon[33795]: osdmap e643: 8 total, 8 up, 8 in 2024-02-10T06:11:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:52 smithi121 ceph-mon[41707]: pgmap v980: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:52 smithi121 ceph-mon[30020]: pgmap v980: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:52 smithi146 ceph-mon[33795]: pgmap v980: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:11:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:54 smithi121 ceph-mon[41707]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-02-10T06:11:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:54 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:54 smithi121 ceph-mon[30020]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-02-10T06:11:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:54 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:54.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:54 smithi146 ceph-mon[33795]: pgmap v981: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 356 B/s wr, 1 op/s 2024-02-10T06:11:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:54 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:11:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:11:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:11:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:11:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[41707]: pgmap v982: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 303 B/s wr, 1 op/s 2024-02-10T06:11:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[30020]: pgmap v982: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 303 B/s wr, 1 op/s 2024-02-10T06:11:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:56.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:56 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:11:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:11:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:56 smithi146 ceph-mon[33795]: pgmap v982: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 303 B/s wr, 1 op/s 2024-02-10T06:11:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:56 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[41707]: osdmap e644: 8 total, 8 up, 8 in 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[30020]: osdmap e644: 8 total, 8 up, 8 in 2024-02-10T06:11:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:11:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:57 smithi146 ceph-mon[33795]: osdmap e644: 8 total, 8 up, 8 in 2024-02-10T06:11:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:57 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]: dispatch 2024-02-10T06:11:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[41707]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[41707]: osdmap e645: 8 total, 8 up, 8 in 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[30020]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-02-10T06:11:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:58 smithi121 ceph-mon[30020]: osdmap e645: 8 total, 8 up, 8 in 2024-02-10T06:11:58.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:58 smithi146 ceph-mon[33795]: pgmap v984: 268 pgs: 268 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 264 B/s wr, 1 op/s 2024-02-10T06:11:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:58 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:11:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:58 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-128"}]': finished 2024-02-10T06:11:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:58 smithi146 ceph-mon[33795]: osdmap e645: 8 total, 8 up, 8 in 2024-02-10T06:11:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:11:59 smithi121 ceph-mon[41707]: osdmap e646: 8 total, 8 up, 8 in 2024-02-10T06:11:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:11:59 smithi121 ceph-mon[30020]: osdmap e646: 8 total, 8 up, 8 in 2024-02-10T06:11:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:11:59 smithi146 ceph-mon[33795]: osdmap e646: 8 total, 8 up, 8 in 2024-02-10T06:12:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[41707]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:12:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[41707]: osdmap e647: 8 total, 8 up, 8 in 2024-02-10T06:12:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[30020]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:12:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[30020]: osdmap e647: 8 total, 8 up, 8 in 2024-02-10T06:12:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:00.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:00 smithi146 ceph-mon[33795]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T06:12:00.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:00 smithi146 ceph-mon[33795]: osdmap e647: 8 total, 8 up, 8 in 2024-02-10T06:12:00.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:00.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[41707]: osdmap e648: 8 total, 8 up, 8 in 2024-02-10T06:12:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[30020]: osdmap e648: 8 total, 8 up, 8 in 2024-02-10T06:12:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:01 smithi146 ceph-mon[33795]: osdmap e648: 8 total, 8 up, 8 in 2024-02-10T06:12:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:12:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: osdmap e649: 8 total, 8 up, 8 in 2024-02-10T06:12:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: osdmap e649: 8 total, 8 up, 8 in 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:12:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: osdmap e649: 8 total, 8 up, 8 in 2024-02-10T06:12:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[41707]: osdmap e650: 8 total, 8 up, 8 in 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[30020]: osdmap e650: 8 total, 8 up, 8 in 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:03 smithi146 ceph-mon[33795]: osdmap e650: 8 total, 8 up, 8 in 2024-02-10T06:12:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]: dispatch 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: pgmap v993: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]': finished 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: osdmap e651: 8 total, 8 up, 8 in 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: pgmap v993: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]': finished 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: osdmap e651: 8 total, 8 up, 8 in 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: pgmap v993: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-130", "mode": "writeback"}]': finished 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: osdmap e651: 8 total, 8 up, 8 in 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[41707]: osdmap e652: 8 total, 8 up, 8 in 2024-02-10T06:12:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:05.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:05.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[30020]: osdmap e652: 8 total, 8 up, 8 in 2024-02-10T06:12:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:05 smithi146 ceph-mon[33795]: osdmap e652: 8 total, 8 up, 8 in 2024-02-10T06:12:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]: dispatch 2024-02-10T06:12:06.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:06 smithi146 ceph-mon[33795]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:06 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:06.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:06 smithi146 ceph-mon[33795]: osdmap e653: 8 total, 8 up, 8 in 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[30020]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[30020]: osdmap e653: 8 total, 8 up, 8 in 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[41707]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-130"}]': finished 2024-02-10T06:12:06.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:06 smithi121 ceph-mon[41707]: osdmap e653: 8 total, 8 up, 8 in 2024-02-10T06:12:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:07 smithi121 ceph-mon[41707]: osdmap e654: 8 total, 8 up, 8 in 2024-02-10T06:12:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:07 smithi121 ceph-mon[30020]: osdmap e654: 8 total, 8 up, 8 in 2024-02-10T06:12:07.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:07 smithi146 ceph-mon[33795]: osdmap e654: 8 total, 8 up, 8 in 2024-02-10T06:12:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:08 smithi146 ceph-mon[33795]: pgmap v999: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:08 smithi146 ceph-mon[33795]: osdmap e655: 8 total, 8 up, 8 in 2024-02-10T06:12:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[30020]: pgmap v999: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[30020]: osdmap e655: 8 total, 8 up, 8 in 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[41707]: pgmap v999: 236 pgs: 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[41707]: osdmap e655: 8 total, 8 up, 8 in 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: osdmap e656: 8 total, 8 up, 8 in 2024-02-10T06:12:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:09.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:09.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:09.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:09.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:09.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:09 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: osdmap e656: 8 total, 8 up, 8 in 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: osdmap e656: 8 total, 8 up, 8 in 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:12:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:09 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:10 smithi146 ceph-mon[33795]: pgmap v1002: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:10 smithi146 ceph-mon[33795]: osdmap e657: 8 total, 8 up, 8 in 2024-02-10T06:12:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[41707]: pgmap v1002: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[41707]: osdmap e657: 8 total, 8 up, 8 in 2024-02-10T06:12:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[30020]: pgmap v1002: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[30020]: osdmap e657: 8 total, 8 up, 8 in 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:11 smithi146 ceph-mon[33795]: osdmap e658: 8 total, 8 up, 8 in 2024-02-10T06:12:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[30020]: osdmap e658: 8 total, 8 up, 8 in 2024-02-10T06:12:12.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:12.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:12.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[41707]: osdmap e658: 8 total, 8 up, 8 in 2024-02-10T06:12:12.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:12.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]: dispatch 2024-02-10T06:12:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:12 smithi146 ceph-mon[33795]: pgmap v1005: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:12 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]': finished 2024-02-10T06:12:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:12 smithi146 ceph-mon[33795]: osdmap e659: 8 total, 8 up, 8 in 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[41707]: pgmap v1005: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]': finished 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[41707]: osdmap e659: 8 total, 8 up, 8 in 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[30020]: pgmap v1005: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-132", "mode": "writeback"}]': finished 2024-02-10T06:12:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:12 smithi121 ceph-mon[30020]: osdmap e659: 8 total, 8 up, 8 in 2024-02-10T06:12:13.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:13 smithi146 ceph-mon[33795]: osdmap e660: 8 total, 8 up, 8 in 2024-02-10T06:12:14.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:13 smithi121 ceph-mon[30020]: osdmap e660: 8 total, 8 up, 8 in 2024-02-10T06:12:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:13 smithi121 ceph-mon[41707]: osdmap e660: 8 total, 8 up, 8 in 2024-02-10T06:12:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:14 smithi146 ceph-mon[33795]: pgmap v1008: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:14 smithi146 ceph-mon[33795]: osdmap e661: 8 total, 8 up, 8 in 2024-02-10T06:12:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[41707]: pgmap v1008: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[41707]: osdmap e661: 8 total, 8 up, 8 in 2024-02-10T06:12:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[30020]: pgmap v1008: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[30020]: osdmap e661: 8 total, 8 up, 8 in 2024-02-10T06:12:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:15 smithi146 ceph-mon[33795]: osdmap e662: 8 total, 8 up, 8 in 2024-02-10T06:12:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[30020]: osdmap e662: 8 total, 8 up, 8 in 2024-02-10T06:12:16.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:16.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:16.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:16.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[41707]: osdmap e662: 8 total, 8 up, 8 in 2024-02-10T06:12:16.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:16.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:16.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:16 smithi146 ceph-mon[33795]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:16 smithi146 ceph-mon[33795]: osdmap e663: 8 total, 8 up, 8 in 2024-02-10T06:12:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[41707]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[41707]: osdmap e663: 8 total, 8 up, 8 in 2024-02-10T06:12:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[30020]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:16 smithi121 ceph-mon[30020]: osdmap e663: 8 total, 8 up, 8 in 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: osdmap e664: 8 total, 8 up, 8 in 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: osdmap e664: 8 total, 8 up, 8 in 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: osdmap e664: 8 total, 8 up, 8 in 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[41707]: pgmap v1014: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[41707]: osdmap e665: 8 total, 8 up, 8 in 2024-02-10T06:12:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[30020]: pgmap v1014: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[30020]: osdmap e665: 8 total, 8 up, 8 in 2024-02-10T06:12:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:18 smithi146 ceph-mon[33795]: pgmap v1014: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:18 smithi146 ceph-mon[33795]: osdmap e665: 8 total, 8 up, 8 in 2024-02-10T06:12:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]: dispatch 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[41707]: osdmap e666: 8 total, 8 up, 8 in 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:19 smithi121 ceph-mon[30020]: osdmap e666: 8 total, 8 up, 8 in 2024-02-10T06:12:20.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:19 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-132"}]': finished 2024-02-10T06:12:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:19 smithi146 ceph-mon[33795]: osdmap e666: 8 total, 8 up, 8 in 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[41707]: pgmap v1017: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[41707]: osdmap e667: 8 total, 8 up, 8 in 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[30020]: pgmap v1017: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:20 smithi121 ceph-mon[30020]: osdmap e667: 8 total, 8 up, 8 in 2024-02-10T06:12:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:20 smithi146 ceph-mon[33795]: pgmap v1017: 268 pgs: 268 active+clean; 582 KiB data, 667 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-10T06:12:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:20 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:20 smithi146 ceph-mon[33795]: osdmap e667: 8 total, 8 up, 8 in 2024-02-10T06:12:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[30020]: osdmap e668: 8 total, 8 up, 8 in 2024-02-10T06:12:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[41707]: osdmap e668: 8 total, 8 up, 8 in 2024-02-10T06:12:22.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:21 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:21 smithi146 ceph-mon[33795]: osdmap e668: 8 total, 8 up, 8 in 2024-02-10T06:12:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:21 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:22.979 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[41707]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:22.979 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[41707]: osdmap e669: 8 total, 8 up, 8 in 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[30020]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[30020]: osdmap e669: 8 total, 8 up, 8 in 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:22.980 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:22 smithi146 ceph-mon[33795]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:22 smithi146 ceph-mon[33795]: osdmap e669: 8 total, 8 up, 8 in 2024-02-10T06:12:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: osdmap e670: 8 total, 8 up, 8 in 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[41707]: osdmap e671: 8 total, 8 up, 8 in 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: osdmap e670: 8 total, 8 up, 8 in 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:23 smithi121 ceph-mon[30020]: osdmap e671: 8 total, 8 up, 8 in 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: osdmap e670: 8 total, 8 up, 8 in 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:23 smithi146 ceph-mon[33795]: osdmap e671: 8 total, 8 up, 8 in 2024-02-10T06:12:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: pgmap v1023: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]': finished 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: osdmap e672: 8 total, 8 up, 8 in 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: pgmap v1023: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]': finished 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: osdmap e672: 8 total, 8 up, 8 in 2024-02-10T06:12:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: pgmap v1023: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]: dispatch 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-134", "mode": "writeback"}]': finished 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: osdmap e672: 8 total, 8 up, 8 in 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[30020]: pgmap v1026: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[30020]: osdmap e673: 8 total, 8 up, 8 in 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[41707]: pgmap v1026: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[41707]: osdmap e673: 8 total, 8 up, 8 in 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:26.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-mon[33795]: pgmap v1026: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 668 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-mon[33795]: osdmap e673: 8 total, 8 up, 8 in 2024-02-10T06:12:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]: dispatch 2024-02-10T06:12:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[41707]: osdmap e674: 8 total, 8 up, 8 in 2024-02-10T06:12:27.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[30020]: osdmap e674: 8 total, 8 up, 8 in 2024-02-10T06:12:27.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:27 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:27 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-134"}]': finished 2024-02-10T06:12:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:27 smithi146 ceph-mon[33795]: osdmap e674: 8 total, 8 up, 8 in 2024-02-10T06:12:27.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[41707]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T06:12:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[41707]: osdmap e675: 8 total, 8 up, 8 in 2024-02-10T06:12:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[30020]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T06:12:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:28 smithi121 ceph-mon[30020]: osdmap e675: 8 total, 8 up, 8 in 2024-02-10T06:12:28.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:28 smithi146 ceph-mon[33795]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-02-10T06:12:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:28 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:28 smithi146 ceph-mon[33795]: osdmap e675: 8 total, 8 up, 8 in 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[41707]: osdmap e676: 8 total, 8 up, 8 in 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[30020]: osdmap e676: 8 total, 8 up, 8 in 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:29.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:29 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:29.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:29 smithi146 ceph-mon[33795]: osdmap e676: 8 total, 8 up, 8 in 2024-02-10T06:12:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:29 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:29.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:29 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[41707]: pgmap v1032: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:12:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[41707]: osdmap e677: 8 total, 8 up, 8 in 2024-02-10T06:12:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[30020]: pgmap v1032: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:12:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[30020]: osdmap e677: 8 total, 8 up, 8 in 2024-02-10T06:12:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:30 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:30 smithi146 ceph-mon[33795]: pgmap v1032: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 673 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:12:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:30 smithi146 ceph-mon[33795]: osdmap e677: 8 total, 8 up, 8 in 2024-02-10T06:12:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:30 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:30 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[41707]: osdmap e678: 8 total, 8 up, 8 in 2024-02-10T06:12:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:31.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:31.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[30020]: osdmap e678: 8 total, 8 up, 8 in 2024-02-10T06:12:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:31.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:31 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:31.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:31 smithi146 ceph-mon[33795]: osdmap e678: 8 total, 8 up, 8 in 2024-02-10T06:12:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:31 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:31.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:31 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: pgmap v1035: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: osdmap e679: 8 total, 8 up, 8 in 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: pgmap v1035: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: osdmap e679: 8 total, 8 up, 8 in 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:32.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: pgmap v1035: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: osdmap e679: 8 total, 8 up, 8 in 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]: dispatch 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:12:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]': finished 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[30020]: osdmap e680: 8 total, 8 up, 8 in 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:33.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:33.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]': finished 2024-02-10T06:12:33.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[41707]: osdmap e680: 8 total, 8 up, 8 in 2024-02-10T06:12:33.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:33.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:33 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:33.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:33 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:12:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-136", "mode": "writeback"}]': finished 2024-02-10T06:12:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:33 smithi146 ceph-mon[33795]: osdmap e680: 8 total, 8 up, 8 in 2024-02-10T06:12:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:33.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:33 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: pgmap v1038: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: osdmap e681: 8 total, 8 up, 8 in 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: pgmap v1038: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: osdmap e681: 8 total, 8 up, 8 in 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:34 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: pgmap v1038: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: osdmap e681: 8 total, 8 up, 8 in 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]: dispatch 2024-02-10T06:12:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:34 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[30020]: osdmap e682: 8 total, 8 up, 8 in 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:35.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:35 smithi121 ceph-mon[41707]: osdmap e682: 8 total, 8 up, 8 in 2024-02-10T06:12:35.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:35 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:12:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:35 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-136"}]': finished 2024-02-10T06:12:35.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:35 smithi146 ceph-mon[33795]: osdmap e682: 8 total, 8 up, 8 in 2024-02-10T06:12:36.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:36 smithi146 ceph-mon[33795]: pgmap v1041: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:36.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:36 smithi146 ceph-mon[33795]: osdmap e683: 8 total, 8 up, 8 in 2024-02-10T06:12:36.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:36 smithi121 ceph-mon[30020]: pgmap v1041: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:36 smithi121 ceph-mon[30020]: osdmap e683: 8 total, 8 up, 8 in 2024-02-10T06:12:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:36 smithi121 ceph-mon[41707]: pgmap v1041: 268 pgs: 268 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:12:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:36 smithi121 ceph-mon[41707]: osdmap e683: 8 total, 8 up, 8 in 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: osdmap e684: 8 total, 8 up, 8 in 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:37.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[30020]: osdmap e685: 8 total, 8 up, 8 in 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: osdmap e684: 8 total, 8 up, 8 in 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:37.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:37 smithi121 ceph-mon[41707]: osdmap e685: 8 total, 8 up, 8 in 2024-02-10T06:12:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: osdmap e684: 8 total, 8 up, 8 in 2024-02-10T06:12:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:37 smithi146 ceph-mon[33795]: osdmap e685: 8 total, 8 up, 8 in 2024-02-10T06:12:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: pgmap v1044: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:12:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: osdmap e686: 8 total, 8 up, 8 in 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: pgmap v1044: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: osdmap e686: 8 total, 8 up, 8 in 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: pgmap v1044: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: osdmap e686: 8 total, 8 up, 8 in 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:12:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:12:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[41707]: osdmap e687: 8 total, 8 up, 8 in 2024-02-10T06:12:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:39.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:12:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[30020]: osdmap e687: 8 total, 8 up, 8 in 2024-02-10T06:12:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:39.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:39 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:39.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:39 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:12:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:39 smithi146 ceph-mon[33795]: osdmap e687: 8 total, 8 up, 8 in 2024-02-10T06:12:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:39.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:39 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:12:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[41707]: pgmap v1047: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:12:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:12:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[41707]: osdmap e688: 8 total, 8 up, 8 in 2024-02-10T06:12:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[30020]: pgmap v1047: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[30020]: osdmap e688: 8 total, 8 up, 8 in 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:40 smithi146 ceph-mon[33795]: pgmap v1047: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-10T06:12:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:12:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:40 smithi146 ceph-mon[33795]: osdmap e688: 8 total, 8 up, 8 in 2024-02-10T06:12:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-10T06:12:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[41707]: pgmap v1050: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:12:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[41707]: osdmap e689: 8 total, 8 up, 8 in 2024-02-10T06:12:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[30020]: pgmap v1050: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:12:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:42 smithi121 ceph-mon[30020]: osdmap e689: 8 total, 8 up, 8 in 2024-02-10T06:12:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:42 smithi146 ceph-mon[33795]: pgmap v1050: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-10T06:12:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:42 smithi146 ceph-mon[33795]: osdmap e689: 8 total, 8 up, 8 in 2024-02-10T06:12:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:43.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:43.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:12:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:43.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]: dispatch 2024-02-10T06:12:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[41707]: pgmap v1052: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]': finished 2024-02-10T06:12:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[41707]: osdmap e690: 8 total, 8 up, 8 in 2024-02-10T06:12:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[30020]: pgmap v1052: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]': finished 2024-02-10T06:12:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[30020]: osdmap e690: 8 total, 8 up, 8 in 2024-02-10T06:12:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:44 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:44 smithi146 ceph-mon[33795]: pgmap v1052: 268 pgs: 268 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-138"}]': finished 2024-02-10T06:12:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:44 smithi146 ceph-mon[33795]: osdmap e690: 8 total, 8 up, 8 in 2024-02-10T06:12:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:44 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:12:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:45.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:45 smithi121 ceph-mon[41707]: osdmap e691: 8 total, 8 up, 8 in 2024-02-10T06:12:45.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:45 smithi121 ceph-mon[30020]: osdmap e691: 8 total, 8 up, 8 in 2024-02-10T06:12:45.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:45 smithi146 ceph-mon[33795]: osdmap e691: 8 total, 8 up, 8 in 2024-02-10T06:12:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[41707]: pgmap v1055: 236 pgs: 236 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[41707]: osdmap e692: 8 total, 8 up, 8 in 2024-02-10T06:12:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[30020]: pgmap v1055: 236 pgs: 236 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[30020]: osdmap e692: 8 total, 8 up, 8 in 2024-02-10T06:12:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:46.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:46 smithi146 ceph-mon[33795]: pgmap v1055: 236 pgs: 236 active+clean; 4.4 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:46 smithi146 ceph-mon[33795]: osdmap e692: 8 total, 8 up, 8 in 2024-02-10T06:12:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:12:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: osdmap e693: 8 total, 8 up, 8 in 2024-02-10T06:12:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: osdmap e693: 8 total, 8 up, 8 in 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:47.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: osdmap e693: 8 total, 8 up, 8 in 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:12:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[41707]: pgmap v1058: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[41707]: osdmap e694: 8 total, 8 up, 8 in 2024-02-10T06:12:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[30020]: pgmap v1058: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[30020]: osdmap e694: 8 total, 8 up, 8 in 2024-02-10T06:12:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:48 smithi146 ceph-mon[33795]: pgmap v1058: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:12:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:48 smithi146 ceph-mon[33795]: osdmap e694: 8 total, 8 up, 8 in 2024-02-10T06:12:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]: dispatch 2024-02-10T06:12:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:12:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[41707]: osdmap e695: 8 total, 8 up, 8 in 2024-02-10T06:12:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:49.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:12:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[30020]: osdmap e695: 8 total, 8 up, 8 in 2024-02-10T06:12:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:49.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:49.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_count","val": "3"}]': finished 2024-02-10T06:12:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:49 smithi146 ceph-mon[33795]: osdmap e695: 8 total, 8 up, 8 in 2024-02-10T06:12:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:49 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:49.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]: dispatch 2024-02-10T06:12:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[41707]: pgmap v1061: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[41707]: osdmap e696: 8 total, 8 up, 8 in 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[30020]: pgmap v1061: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[30020]: osdmap e696: 8 total, 8 up, 8 in 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:50.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:50 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:50 smithi146 ceph-mon[33795]: pgmap v1061: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_period","val": "3"}]': finished 2024-02-10T06:12:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:50 smithi146 ceph-mon[33795]: osdmap e696: 8 total, 8 up, 8 in 2024-02-10T06:12:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:50 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:50 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:12:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:12:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[30020]: osdmap e697: 8 total, 8 up, 8 in 2024-02-10T06:12:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:51.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:51.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:12:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[41707]: osdmap e697: 8 total, 8 up, 8 in 2024-02-10T06:12:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:51.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:51 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:51.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:12:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:51 smithi146 ceph-mon[33795]: osdmap e697: 8 total, 8 up, 8 in 2024-02-10T06:12:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:51 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:51.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:51 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[30020]: pgmap v1064: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[30020]: osdmap e698: 8 total, 8 up, 8 in 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[41707]: pgmap v1064: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:12:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:52 smithi121 ceph-mon[41707]: osdmap e698: 8 total, 8 up, 8 in 2024-02-10T06:12:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:52 smithi146 ceph-mon[33795]: pgmap v1064: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:12:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:52 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-10T06:12:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:52 smithi146 ceph-mon[33795]: osdmap e698: 8 total, 8 up, 8 in 2024-02-10T06:12:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:54 smithi121 ceph-mon[41707]: pgmap v1066: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:54.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:54 smithi121 ceph-mon[30020]: pgmap v1066: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:54 smithi146 ceph-mon[33795]: pgmap v1066: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T06:12:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:12:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:12:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:12:56.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:56 smithi146 ceph-mon[33795]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:12:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:12:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:56 smithi121 ceph-mon[41707]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:56 smithi121 ceph-mon[30020]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:12:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:12:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:12:58 smithi146 ceph-mon[33795]: pgmap v1068: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-02-10T06:12:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:12:58 smithi121 ceph-mon[41707]: pgmap v1068: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-02-10T06:12:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:12:58 smithi121 ceph-mon[30020]: pgmap v1068: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 5.4 KiB/s wr, 2 op/s 2024-02-10T06:13:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:00 smithi146 ceph-mon[33795]: pgmap v1069: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-02-10T06:13:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:00 smithi121 ceph-mon[41707]: pgmap v1069: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-02-10T06:13:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:00 smithi121 ceph-mon[30020]: pgmap v1069: 268 pgs: 268 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-02-10T06:13:02.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:02 smithi146 ceph-mon[33795]: pgmap v1070: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:02 smithi121 ceph-mon[41707]: pgmap v1070: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:02 smithi121 ceph-mon[30020]: pgmap v1070: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:04 smithi146 ceph-mon[33795]: pgmap v1071: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 7.1 KiB/s wr, 1 op/s 2024-02-10T06:13:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[41707]: pgmap v1071: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 7.1 KiB/s wr, 1 op/s 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[30020]: pgmap v1071: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 879 B/s rd, 7.1 KiB/s wr, 1 op/s 2024-02-10T06:13:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]: dispatch 2024-02-10T06:13:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:05.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]': finished 2024-02-10T06:13:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:05 smithi146 ceph-mon[33795]: osdmap e699: 8 total, 8 up, 8 in 2024-02-10T06:13:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]': finished 2024-02-10T06:13:06.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:05 smithi121 ceph-mon[30020]: osdmap e699: 8 total, 8 up, 8 in 2024-02-10T06:13:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-140"}]': finished 2024-02-10T06:13:06.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:05 smithi121 ceph-mon[41707]: osdmap e699: 8 total, 8 up, 8 in 2024-02-10T06:13:06.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:06 smithi146 ceph-mon[33795]: pgmap v1073: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:06 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:06 smithi146 ceph-mon[33795]: osdmap e700: 8 total, 8 up, 8 in 2024-02-10T06:13:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[41707]: pgmap v1073: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[41707]: osdmap e700: 8 total, 8 up, 8 in 2024-02-10T06:13:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[30020]: pgmap v1073: 268 pgs: 268 active+clean; 4.5 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-02-10T06:13:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:06 smithi121 ceph-mon[30020]: osdmap e700: 8 total, 8 up, 8 in 2024-02-10T06:13:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:07 smithi146 ceph-mon[33795]: osdmap e701: 8 total, 8 up, 8 in 2024-02-10T06:13:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:07 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:07 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[41707]: osdmap e701: 8 total, 8 up, 8 in 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[30020]: osdmap e701: 8 total, 8 up, 8 in 2024-02-10T06:13:08.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:08.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:08.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:08 smithi146 ceph-mon[33795]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:13:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:08 smithi146 ceph-mon[33795]: osdmap e702: 8 total, 8 up, 8 in 2024-02-10T06:13:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[41707]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:13:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[41707]: osdmap e702: 8 total, 8 up, 8 in 2024-02-10T06:13:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[30020]: pgmap v1076: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:13:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[30020]: osdmap e702: 8 total, 8 up, 8 in 2024-02-10T06:13:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:09.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:09 smithi146 ceph-mon[33795]: osdmap e703: 8 total, 8 up, 8 in 2024-02-10T06:13:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:09 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[41707]: osdmap e703: 8 total, 8 up, 8 in 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[30020]: osdmap e703: 8 total, 8 up, 8 in 2024-02-10T06:13:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:10.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:09 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: pgmap v1079: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: osdmap e704: 8 total, 8 up, 8 in 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:10 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: pgmap v1079: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: osdmap e704: 8 total, 8 up, 8 in 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: pgmap v1079: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: osdmap e704: 8 total, 8 up, 8 in 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]: dispatch 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:13:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:13:11.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:10 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]': finished 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: osdmap e705: 8 total, 8 up, 8 in 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]': finished 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: osdmap e705: 8 total, 8 up, 8 in 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:12.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-142", "mode": "writeback"}]': finished 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: osdmap e705: 8 total, 8 up, 8 in 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:12.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:12 smithi146 ceph-mon[33795]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:12 smithi146 ceph-mon[33795]: osdmap e706: 8 total, 8 up, 8 in 2024-02-10T06:13:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[41707]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[41707]: osdmap e706: 8 total, 8 up, 8 in 2024-02-10T06:13:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[30020]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[30020]: osdmap e706: 8 total, 8 up, 8 in 2024-02-10T06:13:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:13 smithi146 ceph-mon[33795]: osdmap e707: 8 total, 8 up, 8 in 2024-02-10T06:13:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:13 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:13 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:13 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[41707]: osdmap e707: 8 total, 8 up, 8 in 2024-02-10T06:13:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:14.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:14.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[30020]: osdmap e707: 8 total, 8 up, 8 in 2024-02-10T06:13:14.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:14.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:14.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:14.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:13 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: osdmap e708: 8 total, 8 up, 8 in 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:14 smithi146 ceph-mon[33795]: osdmap e709: 8 total, 8 up, 8 in 2024-02-10T06:13:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: osdmap e708: 8 total, 8 up, 8 in 2024-02-10T06:13:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[41707]: osdmap e709: 8 total, 8 up, 8 in 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: osdmap e708: 8 total, 8 up, 8 in 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:14 smithi121 ceph-mon[30020]: osdmap e709: 8 total, 8 up, 8 in 2024-02-10T06:13:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:16.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-10T06:13:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: pgmap v1088: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:13:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: osdmap e710: 8 total, 8 up, 8 in 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: pgmap v1088: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: osdmap e710: 8 total, 8 up, 8 in 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: pgmap v1088: 268 pgs: 268 active+clean; 4.4 MiB data, 730 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: osdmap e710: 8 total, 8 up, 8 in 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]: dispatch 2024-02-10T06:13:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]: dispatch 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]': finished 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]': finished 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: osdmap e711: 8 total, 8 up, 8 in 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]': finished 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]': finished 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: osdmap e711: 8 total, 8 up, 8 in 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-10T06:13:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.2", "id": [1, 0]}]': finished 2024-02-10T06:13:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "328.10", "id": [1, 0]}]': finished 2024-02-10T06:13:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: osdmap e711: 8 total, 8 up, 8 in 2024-02-10T06:13:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[30020]: pgmap v1091: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[30020]: osdmap e712: 8 total, 8 up, 8 in 2024-02-10T06:13:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[41707]: pgmap v1091: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[41707]: osdmap e712: 8 total, 8 up, 8 in 2024-02-10T06:13:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:19.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:18 smithi146 ceph-mon[33795]: pgmap v1091: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:18 smithi146 ceph-mon[33795]: osdmap e712: 8 total, 8 up, 8 in 2024-02-10T06:13:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[30020]: osdmap e713: 8 total, 8 up, 8 in 2024-02-10T06:13:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:20.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:20.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[41707]: osdmap e713: 8 total, 8 up, 8 in 2024-02-10T06:13:20.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:20.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:19 smithi146 ceph-mon[33795]: osdmap e713: 8 total, 8 up, 8 in 2024-02-10T06:13:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:19 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: osdmap e714: 8 total, 8 up, 8 in 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: osdmap e714: 8 total, 8 up, 8 in 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:20 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 731 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]': finished 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: osdmap e714: 8 total, 8 up, 8 in 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:20 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-142"}]: dispatch 2024-02-10T06:13:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:21 smithi121 ceph-mon[41707]: osdmap e715: 8 total, 8 up, 8 in 2024-02-10T06:13:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:21 smithi121 ceph-mon[30020]: osdmap e715: 8 total, 8 up, 8 in 2024-02-10T06:13:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:21 smithi146 ceph-mon[33795]: osdmap e715: 8 total, 8 up, 8 in 2024-02-10T06:13:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[30020]: pgmap v1097: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[30020]: osdmap e716: 8 total, 8 up, 8 in 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[41707]: pgmap v1097: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[41707]: osdmap e716: 8 total, 8 up, 8 in 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:22 smithi146 ceph-mon[33795]: pgmap v1097: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:23.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:22 smithi146 ceph-mon[33795]: osdmap e716: 8 total, 8 up, 8 in 2024-02-10T06:13:23.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:23.223 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[30020]: osdmap e717: 8 total, 8 up, 8 in 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[41707]: osdmap e717: 8 total, 8 up, 8 in 2024-02-10T06:13:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:23 smithi146 ceph-mon[33795]: osdmap e717: 8 total, 8 up, 8 in 2024-02-10T06:13:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[30020]: pgmap v1100: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[30020]: osdmap e718: 8 total, 8 up, 8 in 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[41707]: pgmap v1100: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[41707]: osdmap e718: 8 total, 8 up, 8 in 2024-02-10T06:13:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:24 smithi146 ceph-mon[33795]: pgmap v1100: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:13:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:24 smithi146 ceph-mon[33795]: osdmap e718: 8 total, 8 up, 8 in 2024-02-10T06:13:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: osdmap e719: 8 total, 8 up, 8 in 2024-02-10T06:13:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]': finished 2024-02-10T06:13:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[30020]: osdmap e720: 8 total, 8 up, 8 in 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: osdmap e719: 8 total, 8 up, 8 in 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:26.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]': finished 2024-02-10T06:13:26.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:25 smithi121 ceph-mon[41707]: osdmap e720: 8 total, 8 up, 8 in 2024-02-10T06:13:26.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: osdmap e719: 8 total, 8 up, 8 in 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]: dispatch 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-144", "mode": "readproxy"}]': finished 2024-02-10T06:13:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:25 smithi146 ceph-mon[33795]: osdmap e720: 8 total, 8 up, 8 in 2024-02-10T06:13:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:26 smithi121 ceph-mon[30020]: pgmap v1103: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:26 smithi121 ceph-mon[41707]: pgmap v1103: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:27.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:26 smithi146 ceph-mon[33795]: pgmap v1103: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail 2024-02-10T06:13:27.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:28 smithi121 ceph-mon[41707]: pgmap v1105: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 227 B/s wr, 1 op/s 2024-02-10T06:13:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:28 smithi121 ceph-mon[30020]: pgmap v1105: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 227 B/s wr, 1 op/s 2024-02-10T06:13:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:28 smithi146 ceph-mon[33795]: pgmap v1105: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 227 B/s wr, 1 op/s 2024-02-10T06:13:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:30 smithi121 ceph-mon[41707]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:13:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:30 smithi121 ceph-mon[30020]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:13:31.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:30 smithi146 ceph-mon[33795]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-02-10T06:13:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[41707]: pgmap v1107: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 136 B/s wr, 1 op/s 2024-02-10T06:13:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[30020]: pgmap v1107: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 136 B/s wr, 1 op/s 2024-02-10T06:13:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:32 smithi146 ceph-mon[33795]: pgmap v1107: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 136 B/s wr, 1 op/s 2024-02-10T06:13:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:13:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:34 smithi121 ceph-mon[41707]: pgmap v1108: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 120 B/s wr, 1 op/s 2024-02-10T06:13:35.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:34 smithi121 ceph-mon[30020]: pgmap v1108: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 120 B/s wr, 1 op/s 2024-02-10T06:13:35.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:34 smithi146 ceph-mon[33795]: pgmap v1108: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 120 B/s wr, 1 op/s 2024-02-10T06:13:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:36 smithi146 ceph-mon[33795]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:13:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:37.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[41707]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[30020]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: osdmap e721: 8 total, 8 up, 8 in 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: pgmap v1111: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:37 smithi146 ceph-mon[33795]: osdmap e722: 8 total, 8 up, 8 in 2024-02-10T06:13:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: osdmap e721: 8 total, 8 up, 8 in 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: pgmap v1111: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[41707]: osdmap e722: 8 total, 8 up, 8 in 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: osdmap e721: 8 total, 8 up, 8 in 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: pgmap v1111: 268 pgs: 268 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:13:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]': finished 2024-02-10T06:13:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:37 smithi121 ceph-mon[30020]: osdmap e722: 8 total, 8 up, 8 in 2024-02-10T06:13:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:38 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:38 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-144"}]: dispatch 2024-02-10T06:13:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:39 smithi146 ceph-mon[33795]: osdmap e723: 8 total, 8 up, 8 in 2024-02-10T06:13:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:39 smithi146 ceph-mon[33795]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:39 smithi121 ceph-mon[41707]: osdmap e723: 8 total, 8 up, 8 in 2024-02-10T06:13:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:39 smithi121 ceph-mon[41707]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:39 smithi121 ceph-mon[30020]: osdmap e723: 8 total, 8 up, 8 in 2024-02-10T06:13:40.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:39 smithi121 ceph-mon[30020]: pgmap v1114: 236 pgs: 236 active+clean; 4.4 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:13:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:40 smithi146 ceph-mon[33795]: osdmap e724: 8 total, 8 up, 8 in 2024-02-10T06:13:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:40 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[41707]: osdmap e724: 8 total, 8 up, 8 in 2024-02-10T06:13:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[30020]: osdmap e724: 8 total, 8 up, 8 in 2024-02-10T06:13:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:40 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:13:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:41 smithi146 ceph-mon[33795]: osdmap e725: 8 total, 8 up, 8 in 2024-02-10T06:13:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:41 smithi146 ceph-mon[33795]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[41707]: osdmap e725: 8 total, 8 up, 8 in 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[41707]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:42.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:13:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[30020]: osdmap e725: 8 total, 8 up, 8 in 2024-02-10T06:13:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:13:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:41 smithi121 ceph-mon[30020]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:42 smithi146 ceph-mon[33795]: osdmap e726: 8 total, 8 up, 8 in 2024-02-10T06:13:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[41707]: osdmap e726: 8 total, 8 up, 8 in 2024-02-10T06:13:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:13:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[30020]: osdmap e726: 8 total, 8 up, 8 in 2024-02-10T06:13:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: osdmap e727: 8 total, 8 up, 8 in 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]': finished 2024-02-10T06:13:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:43 smithi146 ceph-mon[33795]: osdmap e728: 8 total, 8 up, 8 in 2024-02-10T06:13:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:13:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: osdmap e727: 8 total, 8 up, 8 in 2024-02-10T06:13:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]': finished 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[41707]: osdmap e728: 8 total, 8 up, 8 in 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi121-81810-111", "overlaypool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: osdmap e727: 8 total, 8 up, 8 in 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]: dispatch 2024-02-10T06:13:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-10T06:13:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-10T06:13:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi121-81810-146", "mode": "writeback"}]': finished 2024-02-10T06:13:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:43 smithi121 ceph-mon[30020]: osdmap e728: 8 total, 8 up, 8 in 2024-02-10T06:13:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]: dispatch 2024-02-10T06:13:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:45 smithi146 ceph-mon[33795]: osdmap e729: 8 total, 8 up, 8 in 2024-02-10T06:13:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:45 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:45 smithi146 ceph-mon[33795]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:13:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[41707]: osdmap e729: 8 total, 8 up, 8 in 2024-02-10T06:13:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[41707]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:13:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_count","val": "2"}]': finished 2024-02-10T06:13:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[30020]: osdmap e729: 8 total, 8 up, 8 in 2024-02-10T06:13:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]: dispatch 2024-02-10T06:13:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:45 smithi121 ceph-mon[30020]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:13:47.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: osdmap e730: 8 total, 8 up, 8 in 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:46 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: osdmap e730: 8 total, 8 up, 8 in 2024-02-10T06:13:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_period","val": "600"}]': finished 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: osdmap e730: 8 total, 8 up, 8 in 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:46 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:13:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: osdmap e731: 8 total, 8 up, 8 in 2024-02-10T06:13:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[41707]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: osdmap e731: 8 total, 8 up, 8 in 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:47 smithi121 ceph-mon[30020]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "hit_set_type","val": "bloom"}]': finished 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: osdmap e731: 8 total, 8 up, 8 in 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-10T06:13:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:47 smithi146 ceph-mon[33795]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[41707]: osdmap e732: 8 total, 8 up, 8 in 2024-02-10T06:13:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[30020]: osdmap e732: 8 total, 8 up, 8 in 2024-02-10T06:13:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:48 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-10T06:13:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:48 smithi146 ceph-mon[33795]: osdmap e732: 8 total, 8 up, 8 in 2024-02-10T06:13:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:48 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:48 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]: dispatch 2024-02-10T06:13:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:13:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[41707]: osdmap e733: 8 total, 8 up, 8 in 2024-02-10T06:13:50.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[41707]: pgmap v1129: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:50.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:13:50.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[30020]: osdmap e733: 8 total, 8 up, 8 in 2024-02-10T06:13:50.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:49 smithi121 ceph-mon[30020]: pgmap v1129: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:50.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:49 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-146","var": "target_max_objects","val": "1"}]': finished 2024-02-10T06:13:50.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:49 smithi146 ceph-mon[33795]: osdmap e733: 8 total, 8 up, 8 in 2024-02-10T06:13:50.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:49 smithi146 ceph-mon[33795]: pgmap v1129: 268 pgs: 268 active+clean; 4.4 MiB data, 755 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:51.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:50 smithi121 ceph-mon[30020]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:13:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:50 smithi121 ceph-mon[41707]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:13:51.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:50 smithi146 ceph-mon[33795]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-10T06:13:52.229 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:51 smithi121 ceph-mon[41707]: pgmap v1130: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:52.230 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:51 smithi121 ceph-mon[30020]: pgmap v1130: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:52.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:51 smithi146 ceph-mon[33795]: pgmap v1130: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-02-10T06:13:54.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:54 smithi121 ceph-mon[30020]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:54.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:54 smithi121 ceph-mon[41707]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:54.671 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:54 smithi146 ceph-mon[33795]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 810 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:13:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:13:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:13:56.743 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:56 smithi146 ceph-mon[33795]: pgmap v1132: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:13:56.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:56 smithi121 ceph-mon[41707]: pgmap v1132: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:13:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:56 smithi121 ceph-mon[30020]: pgmap v1132: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-02-10T06:13:57.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:13:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:13:57.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:57.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:57.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:13:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:58 smithi121 ceph-mon[41707]: pgmap v1133: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:58 smithi121 ceph-mon[30020]: pgmap v1133: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:58 smithi146 ceph-mon[33795]: pgmap v1133: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:13:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:59 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:59.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:13:59 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:59 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:59.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:13:59 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:59.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:59 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:13:59.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:13:59 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[41707]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[41707]: osdmap e734: 8 total, 8 up, 8 in 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[30020]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[30020]: osdmap e734: 8 total, 8 up, 8 in 2024-02-10T06:14:00.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:00.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:00 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:00 smithi146 ceph-mon[33795]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 993 B/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:00 smithi146 ceph-mon[33795]: osdmap e734: 8 total, 8 up, 8 in 2024-02-10T06:14:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:00 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:14:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: osdmap e735: 8 total, 8 up, 8 in 2024-02-10T06:14:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: osdmap e735: 8 total, 8 up, 8 in 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:01 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]': finished 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: osdmap e735: 8 total, 8 up, 8 in 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:01.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:01 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-146"}]: dispatch 2024-02-10T06:14:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: osdmap e736: 8 total, 8 up, 8 in 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:14:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: osdmap e736: 8 total, 8 up, 8 in 2024-02-10T06:14:02.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:02.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: osdmap e736: 8 total, 8 up, 8 in 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[41707]: osdmap e737: 8 total, 8 up, 8 in 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[30020]: osdmap e737: 8 total, 8 up, 8 in 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:03.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:03 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:03 smithi146 ceph-mon[33795]: osdmap e737: 8 total, 8 up, 8 in 2024-02-10T06:14:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:03 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:03.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:03 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[41707]: pgmap v1140: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:14:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[41707]: osdmap e738: 8 total, 8 up, 8 in 2024-02-10T06:14:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[30020]: pgmap v1140: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:14:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[30020]: osdmap e738: 8 total, 8 up, 8 in 2024-02-10T06:14:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:04.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:04 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:04 smithi146 ceph-mon[33795]: pgmap v1140: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-02-10T06:14:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:04 smithi146 ceph-mon[33795]: osdmap e738: 8 total, 8 up, 8 in 2024-02-10T06:14:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:04 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:04 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-10T06:14:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:05.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:14:05.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[30020]: osdmap e739: 8 total, 8 up, 8 in 2024-02-10T06:14:05.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:05.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:05.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:14:05.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[41707]: osdmap e739: 8 total, 8 up, 8 in 2024-02-10T06:14:05.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:05.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:05 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:05.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148", "force_nonempty": "--force-nonempty" }]': finished 2024-02-10T06:14:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:05 smithi146 ceph-mon[33795]: osdmap e739: 8 total, 8 up, 8 in 2024-02-10T06:14:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:05 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:05.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:05 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: pgmap v1143: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]': finished 2024-02-10T06:14:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: osdmap e740: 8 total, 8 up, 8 in 2024-02-10T06:14:06.823 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.824 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:06 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: pgmap v1143: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]': finished 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: osdmap e740: 8 total, 8 up, 8 in 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: pgmap v1143: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 756 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]': finished 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: osdmap e740: 8 total, 8 up, 8 in 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:06.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:06 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-148"}]: dispatch 2024-02-10T06:14:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:07 smithi121 ceph-mon[41707]: osdmap e741: 8 total, 8 up, 8 in 2024-02-10T06:14:07.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:07 smithi121 ceph-mon[30020]: osdmap e741: 8 total, 8 up, 8 in 2024-02-10T06:14:07.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:07 smithi146 ceph-mon[33795]: osdmap e741: 8 total, 8 up, 8 in 2024-02-10T06:14:07.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[41707]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[41707]: osdmap e742: 8 total, 8 up, 8 in 2024-02-10T06:14:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[30020]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[30020]: osdmap e742: 8 total, 8 up, 8 in 2024-02-10T06:14:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:08 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:08 smithi146 ceph-mon[33795]: pgmap v1146: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:08 smithi146 ceph-mon[33795]: osdmap e742: 8 total, 8 up, 8 in 2024-02-10T06:14:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:08 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:08 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:09.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:09.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: osdmap e743: 8 total, 8 up, 8 in 2024-02-10T06:14:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[30020]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: osdmap e743: 8 total, 8 up, 8 in 2024-02-10T06:14:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:09 smithi121 ceph-mon[41707]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: osdmap e743: 8 total, 8 up, 8 in 2024-02-10T06:14:09.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-150"}]: dispatch 2024-02-10T06:14:09.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:09 smithi146 ceph-mon[33795]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:10 smithi121 ceph-mon[30020]: pgmap v1149: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:10 smithi121 ceph-mon[30020]: osdmap e744: 8 total, 8 up, 8 in 2024-02-10T06:14:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:10 smithi121 ceph-mon[41707]: pgmap v1149: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:10 smithi121 ceph-mon[41707]: osdmap e744: 8 total, 8 up, 8 in 2024-02-10T06:14:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:10 smithi146 ceph-mon[33795]: pgmap v1149: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-10T06:14:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:10 smithi146 ceph-mon[33795]: osdmap e744: 8 total, 8 up, 8 in 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: osdmap e745: 8 total, 8 up, 8 in 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:11.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: osdmap e745: 8 total, 8 up, 8 in 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:11.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:11 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: osdmap e745: 8 total, 8 up, 8 in 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:11.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:11 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: pgmap v1152: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: osdmap e746: 8 total, 8 up, 8 in 2024-02-10T06:14:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[41707]: osdmap e747: 8 total, 8 up, 8 in 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: pgmap v1152: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: osdmap e746: 8 total, 8 up, 8 in 2024-02-10T06:14:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:13.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:12 smithi121 ceph-mon[30020]: osdmap e747: 8 total, 8 up, 8 in 2024-02-10T06:14:13.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=sqlstore.transactions t=2024-02-10T06:14:12.961236004Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T06:14:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: pgmap v1152: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: osdmap e746: 8 total, 8 up, 8 in 2024-02-10T06:14:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-152"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:14:13.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:12 smithi146 ceph-mon[33795]: osdmap e747: 8 total, 8 up, 8 in 2024-02-10T06:14:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[41707]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[41707]: osdmap e748: 8 total, 8 up, 8 in 2024-02-10T06:14:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[30020]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[30020]: osdmap e748: 8 total, 8 up, 8 in 2024-02-10T06:14:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:14 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:14 smithi146 ceph-mon[33795]: pgmap v1155: 236 pgs: 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-10T06:14:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:14 smithi146 ceph-mon[33795]: osdmap e748: 8 total, 8 up, 8 in 2024-02-10T06:14:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:14 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:14 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-10T06:14:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:15.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: osdmap e749: 8 total, 8 up, 8 in 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: osdmap e749: 8 total, 8 up, 8 in 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.830 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:15 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi121-81810-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: osdmap e749: 8 total, 8 up, 8 in 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi121-81810-111","var": "dedup_tier","val": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:15.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:15 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi121-81810-111", "tierpool": "test-rados-api-smithi121-81810-154"}]: dispatch 2024-02-10T06:14:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:16 smithi121 ceph-mon[30020]: pgmap v1158: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:16 smithi121 ceph-mon[30020]: osdmap e750: 8 total, 8 up, 8 in 2024-02-10T06:14:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:16 smithi121 ceph-mon[41707]: pgmap v1158: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:16.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:16 smithi121 ceph-mon[41707]: osdmap e750: 8 total, 8 up, 8 in 2024-02-10T06:14:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:16 smithi146 ceph-mon[33795]: pgmap v1158: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:16 smithi146 ceph-mon[33795]: osdmap e750: 8 total, 8 up, 8 in 2024-02-10T06:14:16.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[30020]: osdmap e751: 8 total, 8 up, 8 in 2024-02-10T06:14:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[41707]: osdmap e751: 8 total, 8 up, 8 in 2024-02-10T06:14:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:17.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:17 smithi146 ceph-mon[33795]: osdmap e751: 8 total, 8 up, 8 in 2024-02-10T06:14:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:18.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7875 ms) 2024-02-10T06:14:18.590 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13071 ms) 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8062 ms) 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13125 ms) 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7743 ms) 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8052 ms) 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-10T06:14:18.591 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: ok, hit_set contains 326:602f83fe:::foo:head 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9017 ms) 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: first is 1707545572 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,0 2024-02-10T06:14:18.592 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,1707545580,1707545581,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,1707545580,1707545581,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545572,1707545574,1707545575,1707545577,1707545578,1707545580,1707545581,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: got ls 1707545575,1707545577,1707545578,1707545580,1707545581,1707545583,1707545584,0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: first now 1707545575, trimmed 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21112 ms) 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: foo0 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15100 ms) 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-10T06:14:18.593 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18126 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22640 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5023 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3026 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3009 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3020 ms) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (233477 ms total) 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-10T06:14:18.594 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (858823 ms total) 2024-02-10T06:14:18.595 INFO:tasks.workunit.client.0.smithi121.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81495 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81495 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81741 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81741 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82199 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82199 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.598 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81992 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81992 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81847 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81847 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81582 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81582 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.599 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81927 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81927 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82280 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82280 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82310 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82310 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81607 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81607 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82222 2024-02-10T06:14:18.600 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82222 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82147 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82147 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81469 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81469 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81533 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81533 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.601 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82161 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82161 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81460 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81460 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81480 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81480 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82454 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82454 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81765 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81765 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.602 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82663 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82663 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81913 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81913 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82264 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82264 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82794 2024-02-10T06:14:18.603 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82794 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: pgmap v1161: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: osdmap e752: 8 total, 8 up, 8 in 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: pgmap v1161: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: osdmap e752: 8 total, 8 up, 8 in 2024-02-10T06:14:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: pgmap v1161: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: osdmap e752: 8 total, 8 up, 8 in 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/892697598' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]: dispatch 2024-02-10T06:14:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:19 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:19.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:19 smithi146 ceph-mon[33795]: osdmap e753: 8 total, 8 up, 8 in 2024-02-10T06:14:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:19 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:20.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:19 smithi121 ceph-mon[30020]: osdmap e753: 8 total, 8 up, 8 in 2024-02-10T06:14:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:19 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi121-81810-111"}]': finished 2024-02-10T06:14:20.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:19 smithi121 ceph-mon[41707]: osdmap e753: 8 total, 8 up, 8 in 2024-02-10T06:14:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:20 smithi146 ceph-mon[33795]: pgmap v1164: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:20 smithi121 ceph-mon[30020]: pgmap v1164: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:20 smithi121 ceph-mon[41707]: pgmap v1164: 228 pgs: 228 active+clean; 582 KiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:22.880 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:22 smithi121 ceph-mon[41707]: pgmap v1165: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:22.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:22 smithi121 ceph-mon[30020]: pgmap v1165: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:22 smithi146 ceph-mon[33795]: pgmap v1165: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:14:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:24 smithi146 ceph-mon[33795]: pgmap v1166: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:14:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:24 smithi121 ceph-mon[41707]: pgmap v1166: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:14:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:24 smithi121 ceph-mon[30020]: pgmap v1166: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:14:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:26 smithi146 ceph-mon[33795]: pgmap v1167: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:14:26.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:26 smithi121 ceph-mon[30020]: pgmap v1167: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:14:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:26 smithi121 ceph-mon[41707]: pgmap v1167: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:14:27.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:28.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:28 smithi146 ceph-mon[33795]: pgmap v1168: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:14:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:28 smithi121 ceph-mon[30020]: pgmap v1168: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:14:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:28 smithi121 ceph-mon[41707]: pgmap v1168: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:14:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:30 smithi146 ceph-mon[33795]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-02-10T06:14:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:30 smithi121 ceph-mon[41707]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-02-10T06:14:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:30 smithi121 ceph-mon[30020]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-02-10T06:14:32.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:32 smithi146 ceph-mon[33795]: pgmap v1170: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:32 smithi121 ceph-mon[41707]: pgmap v1170: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:32 smithi121 ceph-mon[30020]: pgmap v1170: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:34 smithi146 ceph-mon[33795]: pgmap v1171: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:34 smithi121 ceph-mon[30020]: pgmap v1171: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:34 smithi121 ceph-mon[41707]: pgmap v1171: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:36 smithi146 ceph-mon[33795]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:36.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:36 smithi121 ceph-mon[41707]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:36 smithi121 ceph-mon[30020]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:37.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:38.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:38 smithi146 ceph-mon[33795]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:38 smithi121 ceph-mon[30020]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:38 smithi121 ceph-mon[41707]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:40 smithi146 ceph-mon[33795]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:40 smithi121 ceph-mon[30020]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:40 smithi121 ceph-mon[41707]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:42 smithi146 ceph-mon[33795]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:42 smithi121 ceph-mon[41707]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:42 smithi121 ceph-mon[30020]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:44 smithi146 ceph-mon[33795]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:45.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:44 smithi121 ceph-mon[41707]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:44 smithi121 ceph-mon[30020]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:46.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:46 smithi146 ceph-mon[33795]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:46.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:46 smithi121 ceph-mon[41707]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:46 smithi121 ceph-mon[30020]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:14:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:48 smithi146 ceph-mon[33795]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:48 smithi121 ceph-mon[41707]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:48 smithi121 ceph-mon[30020]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:50 smithi146 ceph-mon[33795]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:50 smithi121 ceph-mon[30020]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:50 smithi121 ceph-mon[41707]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:52.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:52 smithi146 ceph-mon[33795]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:52 smithi121 ceph-mon[41707]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:52 smithi121 ceph-mon[30020]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:54 smithi146 ceph-mon[33795]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:54 smithi121 ceph-mon[41707]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:54 smithi121 ceph-mon[30020]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:14:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:14:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:14:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:14:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:56 smithi146 ceph-mon[33795]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:14:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:14:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:56 smithi121 ceph-mon[30020]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:56 smithi121 ceph-mon[41707]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:14:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:58.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:14:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:14:58 smithi121 ceph-mon[30020]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:14:58 smithi121 ceph-mon[41707]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:14:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:14:58 smithi146 ceph-mon[33795]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:00 smithi121 ceph-mon[30020]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:00 smithi121 ceph-mon[41707]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:01.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:00 smithi146 ceph-mon[33795]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:02 smithi121 ceph-mon[30020]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:02 smithi121 ceph-mon[41707]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:03.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:02 smithi146 ceph-mon[33795]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:04 smithi121 ceph-mon[41707]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:04 smithi121 ceph-mon[30020]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:04 smithi146 ceph-mon[33795]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:06.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:06.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:06 smithi146 ceph-mon[33795]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:06 smithi121 ceph-mon[41707]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:06 smithi121 ceph-mon[30020]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:08.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:08.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:08.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:08 smithi121 ceph-mon[41707]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:08 smithi121 ceph-mon[30020]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:08 smithi146 ceph-mon[33795]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:10 smithi121 ceph-mon[41707]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:10 smithi121 ceph-mon[30020]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:10 smithi146 ceph-mon[33795]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[41707]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[30020]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:12 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:15:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:12 smithi146 ceph-mon[33795]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:15:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:15:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:15:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:12 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:15:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:14 smithi121 ceph-mon[41707]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:14 smithi121 ceph-mon[30020]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:15.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:14 smithi146 ceph-mon[33795]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:16 smithi121 ceph-mon[41707]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:17.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:16 smithi121 ceph-mon[30020]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:17.645 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:16 smithi146 ceph-mon[33795]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:17.645 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:18.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:17 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:18.081 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:18.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:17 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:18.081 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:17 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:18 smithi121 ceph-mon[41707]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:18 smithi121 ceph-mon[30020]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:18 smithi146 ceph-mon[33795]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:20 smithi121 ceph-mon[41707]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:20 smithi121 ceph-mon[30020]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:20 smithi146 ceph-mon[33795]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:22 smithi121 ceph-mon[30020]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:22 smithi121 ceph-mon[41707]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:22 smithi146 ceph-mon[33795]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:24 smithi121 ceph-mon[41707]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:24 smithi121 ceph-mon[30020]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:24 smithi146 ceph-mon[33795]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:26 smithi121 ceph-mon[41707]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:26 smithi121 ceph-mon[30020]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:26 smithi146 ceph-mon[33795]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:27.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:28.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:28.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:28.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:28 smithi121 ceph-mon[41707]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:28 smithi121 ceph-mon[30020]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:28 smithi146 ceph-mon[33795]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:30 smithi121 ceph-mon[41707]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:30 smithi121 ceph-mon[30020]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:30 smithi146 ceph-mon[33795]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:32 smithi121 ceph-mon[41707]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:32 smithi121 ceph-mon[30020]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:32 smithi146 ceph-mon[33795]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:35.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:34 smithi121 ceph-mon[41707]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:34 smithi121 ceph-mon[30020]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:34 smithi146 ceph-mon[33795]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:36 smithi121 ceph-mon[41707]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:36 smithi121 ceph-mon[30020]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:36 smithi146 ceph-mon[33795]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:37.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:38.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:37 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:38.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:37 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:38.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:37 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:38 smithi121 ceph-mon[41707]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:38 smithi121 ceph-mon[30020]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:39.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:38 smithi146 ceph-mon[33795]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:40 smithi121 ceph-mon[41707]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:40 smithi121 ceph-mon[30020]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:40 smithi146 ceph-mon[33795]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:43.083 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:42 smithi121 ceph-mon[41707]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:43.083 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:42 smithi121 ceph-mon[30020]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:43.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:42 smithi146 ceph-mon[33795]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:45.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:44 smithi121 ceph-mon[41707]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:45.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:44 smithi121 ceph-mon[30020]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:45.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:45.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:44 smithi146 ceph-mon[33795]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:46 smithi121 ceph-mon[41707]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:46 smithi121 ceph-mon[30020]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:47.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:46 smithi146 ceph-mon[33795]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:47.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:47 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:47 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:47 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:15:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:48 smithi121 ceph-mon[41707]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:48 smithi121 ceph-mon[30020]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:49.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:48 smithi146 ceph-mon[33795]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:50 smithi121 ceph-mon[41707]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:50 smithi121 ceph-mon[30020]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:51.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:50 smithi146 ceph-mon[33795]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:52 smithi121 ceph-mon[41707]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:52 smithi121 ceph-mon[30020]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:52 smithi146 ceph-mon[33795]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:54 smithi121 ceph-mon[41707]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:55.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:54 smithi121 ceph-mon[30020]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:55.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:15:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:15:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:15:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:54 smithi146 ceph-mon[33795]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:15:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:56 smithi121 ceph-mon[30020]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:56 smithi121 ceph-mon[41707]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:56 smithi146 ceph-mon[33795]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:15:57.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:15:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:15:58.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:58.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:15:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:15:58 smithi121 ceph-mon[41707]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:15:58 smithi121 ceph-mon[30020]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:15:59.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:15:58 smithi146 ceph-mon[33795]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:00 smithi121 ceph-mon[30020]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:00 smithi121 ceph-mon[41707]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:00 smithi146 ceph-mon[33795]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:02 smithi121 ceph-mon[30020]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:02 smithi121 ceph-mon[41707]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:02 smithi146 ceph-mon[33795]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:05.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:04 smithi121 ceph-mon[41707]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:04 smithi121 ceph-mon[30020]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:05.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:05.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:04 smithi146 ceph-mon[33795]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:06 smithi121 ceph-mon[41707]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:06 smithi121 ceph-mon[30020]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:07.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:06 smithi146 ceph-mon[33795]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:07.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:08.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:07 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:08.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:07 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:08.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:07 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:09.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:08 smithi146 ceph-mon[33795]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:08 smithi121 ceph-mon[30020]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:08 smithi121 ceph-mon[41707]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:11.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:10 smithi146 ceph-mon[33795]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:10 smithi121 ceph-mon[41707]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:11.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:10 smithi121 ceph-mon[30020]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:12 smithi146 ceph-mon[33795]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:12 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:12 smithi121 ceph-mon[30020]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:12 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:12 smithi121 ceph-mon[41707]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:12 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:16:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:13 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:13 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:13 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:16:14.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:14.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:14.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:16:14.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:16:14.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:16:14.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:13 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:16:15.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:14 smithi146 ceph-mon[33795]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:15.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:14 smithi121 ceph-mon[30020]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:14 smithi121 ceph-mon[41707]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:16 smithi146 ceph-mon[33795]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:17.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:16 smithi121 ceph-mon[30020]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:16 smithi121 ceph-mon[41707]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:19.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:18 smithi146 ceph-mon[33795]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:18 smithi121 ceph-mon[41707]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:18 smithi121 ceph-mon[30020]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:20 smithi146 ceph-mon[33795]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:20 smithi121 ceph-mon[41707]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:20 smithi121 ceph-mon[30020]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:23.130 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:22 smithi121 ceph-mon[41707]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:23.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:22 smithi121 ceph-mon[30020]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:22 smithi146 ceph-mon[33795]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:23 smithi146 ceph-mon[33795]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:24.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:23 smithi121 ceph-mon[30020]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:24.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:23 smithi121 ceph-mon[41707]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:26 smithi121 ceph-mon[41707]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:26 smithi121 ceph-mon[30020]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:26.865 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:26 smithi146 ceph-mon[33795]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:27.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:28.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:28 smithi121 ceph-mon[30020]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:28.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:28 smithi121 ceph-mon[41707]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:28.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:28 smithi146 ceph-mon[33795]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:30 smithi121 ceph-mon[41707]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:30.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:30 smithi121 ceph-mon[30020]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:30 smithi146 ceph-mon[33795]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:32 smithi121 ceph-mon[41707]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:32 smithi121 ceph-mon[30020]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:32.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:32 smithi146 ceph-mon[33795]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:34 smithi121 ceph-mon[30020]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:34 smithi121 ceph-mon[41707]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:34 smithi146 ceph-mon[33795]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:36 smithi121 ceph-mon[41707]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:36.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:36 smithi121 ceph-mon[30020]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:36.877 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:36 smithi146 ceph-mon[33795]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:37.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:38 smithi121 ceph-mon[41707]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:38 smithi121 ceph-mon[30020]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:38.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:38 smithi146 ceph-mon[33795]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:40 smithi121 ceph-mon[41707]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:40.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:40 smithi121 ceph-mon[30020]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:40 smithi146 ceph-mon[33795]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:42.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:42 smithi121 ceph-mon[41707]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:42 smithi121 ceph-mon[30020]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:42 smithi146 ceph-mon[33795]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:44.832 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:44 smithi121 ceph-mon[41707]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:44.890 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:44 smithi121 ceph-mon[30020]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:44 smithi146 ceph-mon[33795]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:46.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:46 smithi121 ceph-mon[41707]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:46 smithi121 ceph-mon[30020]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:46.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:46 smithi146 ceph-mon[33795]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:47.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:47.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:47.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:16:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:48.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:48 smithi121 ceph-mon[41707]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:48.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:48.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:48 smithi121 ceph-mon[30020]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:48.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:48.924 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:48 smithi146 ceph-mon[33795]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:50.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:50 smithi121 ceph-mon[30020]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:50.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:50 smithi121 ceph-mon[41707]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:50 smithi146 ceph-mon[33795]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:52.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:52 smithi121 ceph-mon[30020]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:52.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:52 smithi121 ceph-mon[41707]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:52 smithi146 ceph-mon[33795]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:54.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:54 smithi121 ceph-mon[30020]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:54.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:54 smithi121 ceph-mon[41707]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:54 smithi146 ceph-mon[33795]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:16:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:16:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:16:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:16:56.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:56 smithi121 ceph-mon[30020]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:56.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:56 smithi121 ceph-mon[41707]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:56.889 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:56 smithi146 ceph-mon[33795]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:16:57.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:16:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:16:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:58.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:16:58 smithi121 ceph-mon[41707]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:58.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:58.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:16:58 smithi121 ceph-mon[30020]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:16:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:16:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:16:58 smithi146 ceph-mon[33795]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:00.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:00 smithi121 ceph-mon[41707]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:00.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:00 smithi121 ceph-mon[30020]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:00.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:00 smithi146 ceph-mon[33795]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:02 smithi121 ceph-mon[41707]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:02.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:02.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:02 smithi121 ceph-mon[30020]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:02.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:02 smithi146 ceph-mon[33795]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:04.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:04 smithi121 ceph-mon[41707]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:04.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:04 smithi121 ceph-mon[30020]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:04 smithi146 ceph-mon[33795]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:06.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:06 smithi121 ceph-mon[41707]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:06.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:06 smithi121 ceph-mon[30020]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:06.901 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:06 smithi146 ceph-mon[33795]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:07.171 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:08.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:08 smithi121 ceph-mon[41707]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:08.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:08 smithi121 ceph-mon[30020]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:08 smithi146 ceph-mon[33795]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:10.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:10 smithi121 ceph-mon[41707]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:10.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:10 smithi121 ceph-mon[30020]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:10.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:10 smithi146 ceph-mon[33795]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:12.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:12 smithi121 ceph-mon[41707]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:12.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:12 smithi121 ceph-mon[30020]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:12.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:12 smithi146 ceph-mon[33795]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:13.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:13 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:13.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:13 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:13.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:13 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:17:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[41707]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:14.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:17:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[30020]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:14.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:14 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:17:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:14 smithi146 ceph-mon[33795]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:17:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:17:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:14 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:17:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:16.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:16 smithi121 ceph-mon[30020]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:16.853 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:16 smithi121 ceph-mon[41707]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:16.912 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:16 smithi146 ceph-mon[33795]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:17.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:17.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:17.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:17.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:18.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:18 smithi121 ceph-mon[41707]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:18.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:18.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:18 smithi121 ceph-mon[30020]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:18 smithi146 ceph-mon[33795]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:20.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:20 smithi121 ceph-mon[30020]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:20.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:20 smithi121 ceph-mon[41707]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:20 smithi146 ceph-mon[33795]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:22 smithi121 ceph-mon[41707]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:22.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:22 smithi121 ceph-mon[30020]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:22.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:22 smithi146 ceph-mon[33795]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:24 smithi121 ceph-mon[41707]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:24 smithi121 ceph-mon[30020]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:24.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:24 smithi146 ceph-mon[33795]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:26 smithi146 ceph-mon[33795]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:26 smithi121 ceph-mon[30020]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:27.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:26 smithi121 ceph-mon[41707]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:27.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:28 smithi146 ceph-mon[33795]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:28 smithi121 ceph-mon[41707]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:28 smithi121 ceph-mon[30020]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:30.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:30 smithi146 ceph-mon[33795]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:30 smithi121 ceph-mon[30020]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:30 smithi121 ceph-mon[41707]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:32 smithi146 ceph-mon[33795]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:32 smithi121 ceph-mon[41707]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:32 smithi121 ceph-mon[30020]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:34 smithi146 ceph-mon[33795]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:35.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:34 smithi121 ceph-mon[41707]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:34 smithi121 ceph-mon[30020]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:36 smithi146 ceph-mon[33795]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:36 smithi121 ceph-mon[41707]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:36 smithi121 ceph-mon[30020]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:37.423 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:38 smithi146 ceph-mon[33795]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:38 smithi121 ceph-mon[41707]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:38 smithi121 ceph-mon[30020]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:40.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:40 smithi146 ceph-mon[33795]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:40 smithi121 ceph-mon[41707]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:40 smithi121 ceph-mon[30020]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 774 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:17:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:42 smithi146 ceph-mon[33795]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:42 smithi121 ceph-mon[41707]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:42 smithi121 ceph-mon[30020]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:44 smithi146 ceph-mon[33795]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:44 smithi121 ceph-mon[41707]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:45.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:44 smithi121 ceph-mon[30020]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:46.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:46 smithi146 ceph-mon[33795]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:46 smithi121 ceph-mon[30020]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:46 smithi121 ceph-mon[41707]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:47.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:17:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:48.923 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:48 smithi146 ceph-mon[33795]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:48 smithi121 ceph-mon[30020]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:48 smithi121 ceph-mon[41707]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:50.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:50 smithi146 ceph-mon[33795]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:50 smithi121 ceph-mon[30020]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:50 smithi121 ceph-mon[41707]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:52 smithi146 ceph-mon[33795]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:52 smithi121 ceph-mon[30020]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:52 smithi121 ceph-mon[41707]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:17:54.880 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:54 smithi121 ceph-mon[41707]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:54.881 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:54 smithi121 ceph-mon[30020]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:54 smithi146 ceph-mon[33795]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:17:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:17:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:17:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:56 smithi146 ceph-mon[33795]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:56 smithi121 ceph-mon[41707]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:56 smithi121 ceph-mon[30020]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:17:57.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:17:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:17:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:17:58 smithi146 ceph-mon[33795]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:17:58 smithi121 ceph-mon[41707]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:17:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:17:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:17:58 smithi121 ceph-mon[30020]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:00 smithi146 ceph-mon[33795]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:00 smithi121 ceph-mon[41707]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:00 smithi121 ceph-mon[30020]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:02 smithi146 ceph-mon[33795]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:02 smithi121 ceph-mon[41707]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:02 smithi121 ceph-mon[30020]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:04 smithi146 ceph-mon[33795]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:04 smithi121 ceph-mon[41707]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:04 smithi121 ceph-mon[30020]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:06.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:06 smithi146 ceph-mon[33795]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:06 smithi121 ceph-mon[41707]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:06 smithi121 ceph-mon[30020]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:07.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:08.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:08.930 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:08 smithi146 ceph-mon[33795]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:08 smithi121 ceph-mon[41707]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:08 smithi121 ceph-mon[30020]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:10 smithi146 ceph-mon[33795]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:10 smithi121 ceph-mon[30020]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:10 smithi121 ceph-mon[41707]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:12 smithi146 ceph-mon[33795]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:12 smithi121 ceph-mon[41707]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:12 smithi121 ceph-mon[30020]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:14 smithi146 ceph-mon[33795]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:14 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:18:15.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[41707]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:15.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:15.009 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[30020]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:18:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:14 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:18:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:16.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:16 smithi146 ceph-mon[33795]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:16 smithi121 ceph-mon[41707]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:16 smithi121 ceph-mon[30020]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:17.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:18 smithi146 ceph-mon[33795]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:18 smithi121 ceph-mon[30020]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:18 smithi121 ceph-mon[41707]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:20 smithi121 ceph-mon[30020]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:20 smithi121 ceph-mon[41707]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:20 smithi146 ceph-mon[33795]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:23.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:22 smithi121 ceph-mon[30020]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:22 smithi121 ceph-mon[41707]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:22 smithi146 ceph-mon[33795]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:24 smithi121 ceph-mon[41707]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:24 smithi121 ceph-mon[30020]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:24 smithi146 ceph-mon[33795]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:26.967 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:26 smithi146 ceph-mon[33795]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:26 smithi121 ceph-mon[30020]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:26 smithi121 ceph-mon[41707]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:27.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:26 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:28 smithi121 ceph-mon[41707]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:28 smithi121 ceph-mon[30020]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:28 smithi146 ceph-mon[33795]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:30 smithi121 ceph-mon[41707]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:30 smithi121 ceph-mon[30020]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:30 smithi146 ceph-mon[33795]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:32 smithi121 ceph-mon[41707]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:32 smithi121 ceph-mon[30020]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:32 smithi146 ceph-mon[33795]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:34 smithi121 ceph-mon[41707]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:34 smithi121 ceph-mon[30020]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:34 smithi146 ceph-mon[33795]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:36.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:36 smithi146 ceph-mon[33795]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:36 smithi121 ceph-mon[41707]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:36 smithi121 ceph-mon[30020]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:37.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:36 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:38 smithi121 ceph-mon[30020]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:38 smithi121 ceph-mon[41707]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:38 smithi146 ceph-mon[33795]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:40 smithi121 ceph-mon[41707]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:40 smithi121 ceph-mon[30020]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:40 smithi146 ceph-mon[33795]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:42 smithi121 ceph-mon[30020]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:42 smithi121 ceph-mon[41707]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:42 smithi146 ceph-mon[33795]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:44 smithi121 ceph-mon[30020]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:45.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:44 smithi121 ceph-mon[41707]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:44 smithi146 ceph-mon[33795]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:46.988 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:46 smithi146 ceph-mon[33795]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:46 smithi121 ceph-mon[41707]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:46 smithi121 ceph-mon[30020]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:47.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:48.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:48.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:48.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:18:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:48 smithi121 ceph-mon[41707]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:48 smithi121 ceph-mon[30020]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:48 smithi146 ceph-mon[33795]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:50 smithi121 ceph-mon[41707]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:50 smithi121 ceph-mon[30020]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:50 smithi146 ceph-mon[33795]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:52 smithi121 ceph-mon[41707]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:52 smithi121 ceph-mon[30020]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:53.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:52 smithi146 ceph-mon[33795]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:54 smithi121 ceph-mon[41707]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:54 smithi121 ceph-mon[30020]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:54 smithi146 ceph-mon[33795]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:18:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:18:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:18:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:18:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:56 smithi121 ceph-mon[41707]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:56 smithi121 ceph-mon[30020]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:56 smithi146 ceph-mon[33795]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:18:57.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:18:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:18:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:18:58 smithi121 ceph-mon[41707]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:18:58 smithi121 ceph-mon[30020]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:18:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:18:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:18:58 smithi146 ceph-mon[33795]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:00 smithi121 ceph-mon[41707]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:00 smithi121 ceph-mon[30020]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:00 smithi146 ceph-mon[33795]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:02 smithi121 ceph-mon[41707]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:02 smithi121 ceph-mon[30020]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:02 smithi146 ceph-mon[33795]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:05.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:04 smithi121 ceph-mon[41707]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:04 smithi121 ceph-mon[30020]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:05.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:04 smithi146 ceph-mon[33795]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:06 smithi121 ceph-mon[41707]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:06 smithi121 ceph-mon[30020]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:06 smithi146 ceph-mon[33795]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:07.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:08 smithi121 ceph-mon[41707]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:08 smithi121 ceph-mon[30020]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:08 smithi146 ceph-mon[33795]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:10 smithi121 ceph-mon[41707]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:10 smithi121 ceph-mon[30020]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:10 smithi146 ceph-mon[33795]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:19:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:12 smithi121 ceph-mon[41707]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:12 smithi121 ceph-mon[30020]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:12 smithi146 ceph-mon[33795]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:13.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:19:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=cleanup t=2024-02-10T06:19:13.597409969Z level=info msg="Completed cleanup jobs" duration=6.319654ms 2024-02-10T06:19:15.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:14 smithi121 ceph-mon[41707]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:15.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:14 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:15.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:14 smithi121 ceph-mon[30020]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:15.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:14 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:15.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:15.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:14 smithi146 ceph-mon[33795]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:14 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:19:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:16.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:19:16.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:16.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:16.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:15 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:19:16.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:19:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:19:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:15 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:19:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:16 smithi121 ceph-mon[41707]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:16 smithi121 ceph-mon[30020]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:16 smithi146 ceph-mon[33795]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:17.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:18 smithi121 ceph-mon[30020]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:18 smithi121 ceph-mon[41707]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:19.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:18 smithi146 ceph-mon[33795]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:20 smithi121 ceph-mon[41707]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:20 smithi121 ceph-mon[30020]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:20 smithi146 ceph-mon[33795]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:22 smithi121 ceph-mon[41707]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:22 smithi121 ceph-mon[30020]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:22 smithi146 ceph-mon[33795]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:25.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:24 smithi121 ceph-mon[41707]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:25.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:24 smithi121 ceph-mon[30020]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:25.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:24 smithi146 ceph-mon[33795]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:26 smithi121 ceph-mon[41707]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:26 smithi121 ceph-mon[30020]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:26 smithi146 ceph-mon[33795]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:27.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:28 smithi121 ceph-mon[41707]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:28 smithi121 ceph-mon[30020]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:28 smithi146 ceph-mon[33795]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:30 smithi121 ceph-mon[41707]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:30 smithi121 ceph-mon[30020]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:30 smithi146 ceph-mon[33795]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:32 smithi121 ceph-mon[41707]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:32 smithi121 ceph-mon[30020]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:33.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:32 smithi146 ceph-mon[33795]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:35.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:34 smithi121 ceph-mon[41707]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:35.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:34 smithi121 ceph-mon[30020]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:35.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:34 smithi146 ceph-mon[33795]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:36 smithi121 ceph-mon[41707]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:36 smithi121 ceph-mon[30020]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:36 smithi146 ceph-mon[33795]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:37.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:38 smithi121 ceph-mon[41707]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:38 smithi121 ceph-mon[30020]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:38 smithi146 ceph-mon[33795]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:40 smithi121 ceph-mon[41707]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:41.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:40 smithi121 ceph-mon[30020]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:40 smithi146 ceph-mon[33795]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:42 smithi121 ceph-mon[41707]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:42 smithi121 ceph-mon[30020]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:42 smithi146 ceph-mon[33795]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:44 smithi146 ceph-mon[33795]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:44 smithi121 ceph-mon[41707]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:44 smithi121 ceph-mon[30020]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:46 smithi146 ceph-mon[33795]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:47.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:46 smithi121 ceph-mon[41707]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:47.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:46 smithi121 ceph-mon[30020]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:48.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:19:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:48 smithi146 ceph-mon[33795]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:48 smithi121 ceph-mon[41707]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:48 smithi121 ceph-mon[30020]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:50 smithi146 ceph-mon[33795]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:50 smithi121 ceph-mon[41707]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:51.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:50 smithi121 ceph-mon[30020]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:52 smithi146 ceph-mon[33795]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:52 smithi121 ceph-mon[41707]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:52 smithi121 ceph-mon[30020]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:55.130 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:54 smithi121 ceph-mon[41707]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:55.131 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:54 smithi121 ceph-mon[30020]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:55.131 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:19:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:19:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:19:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:54 smithi146 ceph-mon[33795]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:56 smithi146 ceph-mon[33795]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:57.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:19:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:19:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:56 smithi121 ceph-mon[41707]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:56 smithi121 ceph-mon[30020]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:19:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:19:58 smithi146 ceph-mon[33795]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:19:58 smithi121 ceph-mon[41707]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:19:59.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:19:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:19:58 smithi121 ceph-mon[30020]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:20:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:00 smithi146 ceph-mon[33795]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:20:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[41707]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: application not enabled on pool 'SnapListsmithi121-82837-1' 2024-02-10T06:20:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: application not enabled on pool 'WatchNotifysmithi121-82855-1' 2024-02-10T06:20:01.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:00 smithi121 ceph-mon[30020]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T06:20:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:02 smithi146 ceph-mon[33795]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:03.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:02 smithi121 ceph-mon[30020]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:03.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:02 smithi121 ceph-mon[41707]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:04 smithi146 ceph-mon[33795]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:05.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:04 smithi121 ceph-mon[30020]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:04 smithi121 ceph-mon[41707]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:05.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:06 smithi146 ceph-mon[33795]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:07.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:06 smithi121 ceph-mon[41707]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:07.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:06 smithi121 ceph-mon[30020]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:08 smithi146 ceph-mon[33795]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:08 smithi121 ceph-mon[30020]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:08 smithi121 ceph-mon[41707]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:10 smithi146 ceph-mon[33795]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:11.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:10 smithi121 ceph-mon[30020]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:10 smithi121 ceph-mon[41707]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:12 smithi146 ceph-mon[33795]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:12 smithi121 ceph-mon[41707]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:13.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:12 smithi121 ceph-mon[30020]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:14 smithi146 ceph-mon[33795]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:14 smithi121 ceph-mon[41707]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:14 smithi121 ceph-mon[30020]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:15 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:15 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:16.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:15 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:20:17.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:16 smithi146 ceph-mon[33795]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:16 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:20:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[41707]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:20:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[30020]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:20:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:20:17.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:16 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:20:18.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:18 smithi146 ceph-mon[33795]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:18 smithi121 ceph-mon[41707]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:18 smithi121 ceph-mon[30020]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:20.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:20 smithi121 ceph-mon[41707]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:20.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:20 smithi121 ceph-mon[30020]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:20.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:20 smithi146 ceph-mon[33795]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:22.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:22 smithi121 ceph-mon[41707]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:22.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:22 smithi121 ceph-mon[30020]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:22 smithi146 ceph-mon[33795]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:24.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:24 smithi121 ceph-mon[41707]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:24.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:24 smithi121 ceph-mon[30020]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:24 smithi146 ceph-mon[33795]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:26 smithi121 ceph-mon[30020]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:26 smithi121 ceph-mon[41707]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:26 smithi146 ceph-mon[33795]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:27.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:28.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:28 smithi121 ceph-mon[41707]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:28.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:28 smithi121 ceph-mon[30020]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:28 smithi146 ceph-mon[33795]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:30 smithi121 ceph-mon[41707]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:30 smithi121 ceph-mon[30020]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:30 smithi146 ceph-mon[33795]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:32 smithi121 ceph-mon[41707]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:32.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:32 smithi121 ceph-mon[30020]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:32.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:32 smithi146 ceph-mon[33795]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:34 smithi121 ceph-mon[41707]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:34 smithi121 ceph-mon[30020]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:34 smithi146 ceph-mon[33795]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:36.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:36 smithi121 ceph-mon[41707]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:36.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:36 smithi121 ceph-mon[30020]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:36 smithi146 ceph-mon[33795]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:37.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:38.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:38 smithi121 ceph-mon[41707]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:38.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:38 smithi121 ceph-mon[30020]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:38 smithi146 ceph-mon[33795]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:40.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:40 smithi121 ceph-mon[41707]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:40.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:40 smithi121 ceph-mon[30020]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:40 smithi146 ceph-mon[33795]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:42.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:42 smithi121 ceph-mon[41707]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:42.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:42 smithi121 ceph-mon[30020]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:42 smithi146 ceph-mon[33795]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:44.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:44 smithi121 ceph-mon[41707]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:44.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:44 smithi121 ceph-mon[30020]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:44.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:44 smithi146 ceph-mon[33795]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:46.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:46 smithi121 ceph-mon[41707]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:46.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:46 smithi121 ceph-mon[30020]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:46 smithi146 ceph-mon[33795]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:47.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:20:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:48 smithi146 ceph-mon[33795]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:48 smithi121 ceph-mon[41707]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:48 smithi121 ceph-mon[30020]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:50 smithi146 ceph-mon[33795]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:50 smithi121 ceph-mon[41707]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:50 smithi121 ceph-mon[30020]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:52 smithi146 ceph-mon[33795]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:52 smithi121 ceph-mon[41707]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:52 smithi121 ceph-mon[30020]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:54 smithi146 ceph-mon[33795]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:54 smithi121 ceph-mon[41707]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:54 smithi121 ceph-mon[30020]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:20:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:20:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:20:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:20:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:56 smithi146 ceph-mon[33795]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:56 smithi121 ceph-mon[41707]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:56 smithi121 ceph-mon[30020]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:20:57.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:20:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:20:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:20:58 smithi146 ceph-mon[33795]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:20:58 smithi121 ceph-mon[41707]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:20:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:20:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:20:58 smithi121 ceph-mon[30020]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:00 smithi146 ceph-mon[33795]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:00 smithi121 ceph-mon[30020]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:00 smithi121 ceph-mon[41707]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:02 smithi146 ceph-mon[33795]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:02 smithi121 ceph-mon[41707]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:02 smithi121 ceph-mon[30020]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:04 smithi146 ceph-mon[33795]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:04 smithi121 ceph-mon[41707]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:04 smithi121 ceph-mon[30020]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:06 smithi146 ceph-mon[33795]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:06 smithi121 ceph-mon[41707]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:07.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:06 smithi121 ceph-mon[30020]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:07.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:08 smithi146 ceph-mon[33795]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:08 smithi121 ceph-mon[30020]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:08 smithi121 ceph-mon[41707]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:10 smithi146 ceph-mon[33795]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:10 smithi121 ceph-mon[41707]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:10 smithi121 ceph-mon[30020]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:12 smithi146 ceph-mon[33795]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:12 smithi121 ceph-mon[41707]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:12 smithi121 ceph-mon[30020]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:14 smithi146 ceph-mon[33795]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:14 smithi121 ceph-mon[30020]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:14 smithi121 ceph-mon[41707]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:16 smithi146 ceph-mon[33795]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:16 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:16.950 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:16 smithi121 ceph-mon[41707]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:16.950 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:16 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:16 smithi121 ceph-mon[30020]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:16 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:21:17.421 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:17.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:17.962 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:17.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:18.907 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:18.907 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:18 smithi121 ceph-mon[41707]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:18.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:18.907 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:18 smithi121 ceph-mon[30020]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:18 smithi146 ceph-mon[33795]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:20 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:20 smithi146 ceph-mon[33795]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[41707]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:20.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:20 smithi121 ceph-mon[30020]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:21 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:21 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:21 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:21 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:21.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:21 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:21:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:21:22.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:21 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:21:22.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:22 smithi146 ceph-mon[33795]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:22 smithi121 ceph-mon[41707]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:22 smithi121 ceph-mon[30020]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:24.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:24 smithi146 ceph-mon[33795]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:24 smithi121 ceph-mon[41707]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:25.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:24 smithi121 ceph-mon[30020]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:26.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:26 smithi146 ceph-mon[33795]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:26 smithi121 ceph-mon[41707]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:26 smithi121 ceph-mon[30020]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:27.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:28.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:28 smithi146 ceph-mon[33795]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:28 smithi121 ceph-mon[41707]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:28 smithi121 ceph-mon[30020]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:30 smithi146 ceph-mon[33795]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:30 smithi121 ceph-mon[41707]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:30 smithi121 ceph-mon[30020]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:32 smithi146 ceph-mon[33795]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:32 smithi121 ceph-mon[41707]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:32 smithi121 ceph-mon[30020]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:34 smithi146 ceph-mon[33795]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:34 smithi121 ceph-mon[41707]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:34 smithi121 ceph-mon[30020]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:36.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:36 smithi146 ceph-mon[33795]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:36 smithi121 ceph-mon[41707]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:36 smithi121 ceph-mon[30020]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:37.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:38 smithi146 ceph-mon[33795]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:38 smithi121 ceph-mon[41707]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:38 smithi121 ceph-mon[30020]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:40 smithi146 ceph-mon[33795]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:40 smithi121 ceph-mon[41707]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:40 smithi121 ceph-mon[30020]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:42 smithi146 ceph-mon[33795]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:42 smithi121 ceph-mon[41707]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:42 smithi121 ceph-mon[30020]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:44 smithi146 ceph-mon[33795]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:45.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:44 smithi121 ceph-mon[41707]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:44 smithi121 ceph-mon[30020]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:46 smithi146 ceph-mon[33795]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:46 smithi121 ceph-mon[41707]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:46 smithi121 ceph-mon[30020]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:47.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:47.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:21:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:48 smithi146 ceph-mon[33795]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:48 smithi121 ceph-mon[41707]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:49.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:48 smithi121 ceph-mon[30020]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:50 smithi146 ceph-mon[33795]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:50 smithi121 ceph-mon[41707]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:50 smithi121 ceph-mon[30020]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:52 smithi146 ceph-mon[33795]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:52 smithi121 ceph-mon[41707]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:52 smithi121 ceph-mon[30020]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:54 smithi146 ceph-mon[33795]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:54 smithi121 ceph-mon[41707]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:54 smithi121 ceph-mon[30020]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:21:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:21:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:21:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:21:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:56 smithi146 ceph-mon[33795]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:56 smithi121 ceph-mon[41707]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:56 smithi121 ceph-mon[30020]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:21:57.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:21:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:21:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:21:58 smithi146 ceph-mon[33795]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:21:58 smithi121 ceph-mon[41707]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:21:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:21:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:21:58 smithi121 ceph-mon[30020]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:00 smithi146 ceph-mon[33795]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:00 smithi121 ceph-mon[41707]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:00 smithi121 ceph-mon[30020]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:02 smithi121 ceph-mon[41707]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:02 smithi121 ceph-mon[30020]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:02 smithi146 ceph-mon[33795]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:04 smithi121 ceph-mon[41707]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:04 smithi121 ceph-mon[30020]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:04 smithi146 ceph-mon[33795]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:06.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:06 smithi146 ceph-mon[33795]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:06 smithi121 ceph-mon[41707]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:06 smithi121 ceph-mon[30020]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-10T06:22:07.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:08 smithi121 ceph-mon[41707]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:08 smithi121 ceph-mon[30020]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:08 smithi146 ceph-mon[33795]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:10 smithi121 ceph-mon[41707]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:11.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:10 smithi121 ceph-mon[30020]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:10 smithi146 ceph-mon[33795]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:12 smithi121 ceph-mon[41707]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:12 smithi121 ceph-mon[30020]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:12 smithi146 ceph-mon[33795]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:14 smithi121 ceph-mon[41707]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:14 smithi121 ceph-mon[30020]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:14 smithi146 ceph-mon[33795]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:16 smithi121 ceph-mon[30020]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:16 smithi121 ceph-mon[41707]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:16 smithi146 ceph-mon[33795]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:18 smithi121 ceph-mon[30020]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:18 smithi121 ceph-mon[41707]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:18 smithi146 ceph-mon[33795]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:20 smithi121 ceph-mon[41707]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:20 smithi121 ceph-mon[30020]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:20 smithi146 ceph-mon[33795]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:21 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:21 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:21 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:22:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[41707]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[30020]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:22:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:22 smithi146 ceph-mon[33795]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:22 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:22:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:22 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:22:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:22:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:24 smithi121 ceph-mon[41707]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:24 smithi121 ceph-mon[30020]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:24 smithi146 ceph-mon[33795]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:26.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:26 smithi121 ceph-mon[30020]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:26.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:26 smithi121 ceph-mon[41707]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:26 smithi146 ceph-mon[33795]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:27.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:28 smithi121 ceph-mon[41707]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:28 smithi121 ceph-mon[30020]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:28 smithi146 ceph-mon[33795]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:30 smithi121 ceph-mon[41707]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:30 smithi121 ceph-mon[30020]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:30 smithi146 ceph-mon[33795]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:32 smithi121 ceph-mon[41707]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:32 smithi121 ceph-mon[30020]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:32 smithi146 ceph-mon[33795]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:35.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:34 smithi121 ceph-mon[41707]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:35.012 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:34 smithi121 ceph-mon[30020]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:35.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:34 smithi146 ceph-mon[33795]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:36 smithi121 ceph-mon[41707]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:36 smithi121 ceph-mon[30020]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:36 smithi146 ceph-mon[33795]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:38 smithi121 ceph-mon[41707]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:38 smithi121 ceph-mon[30020]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:38 smithi146 ceph-mon[33795]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:40 smithi121 ceph-mon[41707]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:40 smithi121 ceph-mon[30020]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:40 smithi146 ceph-mon[33795]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:42 smithi121 ceph-mon[41707]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:43.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:42 smithi121 ceph-mon[30020]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:42 smithi146 ceph-mon[33795]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:44 smithi121 ceph-mon[41707]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:45.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:44 smithi121 ceph-mon[30020]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:44 smithi146 ceph-mon[33795]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:46 smithi121 ceph-mon[41707]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:46 smithi121 ceph-mon[30020]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:46 smithi146 ceph-mon[33795]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:47.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:48.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:22:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:48 smithi121 ceph-mon[41707]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:48 smithi121 ceph-mon[30020]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:48 smithi146 ceph-mon[33795]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:50 smithi121 ceph-mon[41707]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:50 smithi121 ceph-mon[30020]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:50 smithi146 ceph-mon[33795]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:52 smithi121 ceph-mon[41707]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:52 smithi121 ceph-mon[30020]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:52 smithi146 ceph-mon[33795]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:54 smithi121 ceph-mon[41707]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:54 smithi121 ceph-mon[30020]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:55.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:22:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:22:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:22:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:54 smithi146 ceph-mon[33795]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:22:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:56 smithi121 ceph-mon[41707]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:56 smithi121 ceph-mon[30020]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:56 smithi146 ceph-mon[33795]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:22:57.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:22:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:22:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:59.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:22:58 smithi121 ceph-mon[30020]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:22:58 smithi121 ceph-mon[41707]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:22:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:22:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:22:58 smithi146 ceph-mon[33795]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:00 smithi121 ceph-mon[41707]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:00 smithi121 ceph-mon[30020]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:00 smithi146 ceph-mon[33795]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:02 smithi121 ceph-mon[41707]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:02 smithi121 ceph-mon[30020]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:02 smithi146 ceph-mon[33795]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:05.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:04 smithi121 ceph-mon[41707]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:04 smithi121 ceph-mon[30020]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:05.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:04 smithi146 ceph-mon[33795]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:06 smithi121 ceph-mon[41707]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:06 smithi121 ceph-mon[30020]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:06 smithi146 ceph-mon[33795]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:07.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:08 smithi121 ceph-mon[41707]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:08 smithi121 ceph-mon[30020]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:08 smithi146 ceph-mon[33795]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:10 smithi121 ceph-mon[41707]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:10 smithi121 ceph-mon[30020]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:10 smithi146 ceph-mon[33795]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:12 smithi121 ceph-mon[30020]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:13.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:12 smithi121 ceph-mon[41707]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:12 smithi146 ceph-mon[33795]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:15.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:14 smithi121 ceph-mon[41707]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:15.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:15.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:14 smithi121 ceph-mon[30020]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:14 smithi146 ceph-mon[33795]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:16 smithi121 ceph-mon[30020]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:16 smithi121 ceph-mon[41707]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:16 smithi146 ceph-mon[33795]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:18 smithi121 ceph-mon[30020]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:18 smithi121 ceph-mon[41707]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:18 smithi146 ceph-mon[33795]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:20 smithi121 ceph-mon[30020]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:21.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:20 smithi121 ceph-mon[41707]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:20 smithi146 ceph-mon[33795]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[41707]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[30020]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:22 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:22 smithi146 ceph-mon[33795]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:22 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:23:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:22 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:23 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:23 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:23:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:23 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:23:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:23 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:23:25.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:24 smithi121 ceph-mon[41707]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:25.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:24 smithi121 ceph-mon[30020]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:25.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:24 smithi146 ceph-mon[33795]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:26 smithi146 ceph-mon[33795]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:26 smithi121 ceph-mon[30020]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:26 smithi121 ceph-mon[41707]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:27.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:28 smithi146 ceph-mon[33795]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:28 smithi121 ceph-mon[41707]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:28 smithi121 ceph-mon[30020]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:30 smithi146 ceph-mon[33795]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:30 smithi121 ceph-mon[41707]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:31.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:30 smithi121 ceph-mon[30020]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:32 smithi146 ceph-mon[33795]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:32 smithi121 ceph-mon[41707]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:32 smithi121 ceph-mon[30020]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:34 smithi146 ceph-mon[33795]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:34 smithi121 ceph-mon[41707]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:34 smithi121 ceph-mon[30020]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:36 smithi146 ceph-mon[33795]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:36 smithi121 ceph-mon[30020]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:37.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:36 smithi121 ceph-mon[41707]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:38 smithi146 ceph-mon[33795]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:38 smithi121 ceph-mon[41707]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:38 smithi121 ceph-mon[30020]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:40 smithi146 ceph-mon[33795]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:41.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:40 smithi121 ceph-mon[30020]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:40 smithi121 ceph-mon[41707]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:42 smithi146 ceph-mon[33795]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:43.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:42 smithi121 ceph-mon[30020]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:42 smithi121 ceph-mon[41707]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:44 smithi146 ceph-mon[33795]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:44 smithi121 ceph-mon[30020]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:44 smithi121 ceph-mon[41707]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:46 smithi146 ceph-mon[33795]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:46 smithi121 ceph-mon[41707]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:47.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:46 smithi121 ceph-mon[30020]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:47.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:48.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:48.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:23:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:48 smithi146 ceph-mon[33795]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:49.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:49.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:48 smithi121 ceph-mon[30020]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:48 smithi121 ceph-mon[41707]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:50 smithi146 ceph-mon[33795]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:51.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:50 smithi121 ceph-mon[30020]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:50 smithi121 ceph-mon[41707]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:23:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:52 smithi146 ceph-mon[33795]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:52 smithi121 ceph-mon[41707]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:52 smithi121 ceph-mon[30020]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:55.131 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:54 smithi121 ceph-mon[41707]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:55.131 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:23:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:23:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:23:55.131 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:54 smithi121 ceph-mon[30020]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:54 smithi146 ceph-mon[33795]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:56 smithi146 ceph-mon[33795]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:56 smithi121 ceph-mon[41707]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:56 smithi121 ceph-mon[30020]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:23:57.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:23:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:23:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:23:58 smithi146 ceph-mon[33795]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:23:58 smithi121 ceph-mon[41707]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:23:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:23:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:23:58 smithi121 ceph-mon[30020]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:00 smithi146 ceph-mon[33795]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:00 smithi121 ceph-mon[41707]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:00 smithi121 ceph-mon[30020]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:02 smithi146 ceph-mon[33795]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:03.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:02 smithi121 ceph-mon[30020]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:03.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:02 smithi121 ceph-mon[41707]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:03.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:04 smithi146 ceph-mon[33795]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:05.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:04 smithi121 ceph-mon[30020]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:04 smithi121 ceph-mon[41707]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:05.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:06 smithi146 ceph-mon[33795]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:07.331 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:06 smithi121 ceph-mon[30020]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:07.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:06 smithi121 ceph-mon[41707]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:07.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:09.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:08 smithi146 ceph-mon[33795]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:09.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:08 smithi121 ceph-mon[41707]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:08 smithi121 ceph-mon[30020]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:10 smithi146 ceph-mon[33795]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:10 smithi121 ceph-mon[41707]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:11.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:10 smithi121 ceph-mon[30020]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:12 smithi146 ceph-mon[33795]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:12 smithi121 ceph-mon[41707]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:12 smithi121 ceph-mon[30020]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:14 smithi146 ceph-mon[33795]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:14 smithi121 ceph-mon[41707]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:14 smithi121 ceph-mon[30020]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:16 smithi146 ceph-mon[33795]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:16 smithi121 ceph-mon[41707]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:16 smithi121 ceph-mon[30020]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:18 smithi146 ceph-mon[33795]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:18 smithi121 ceph-mon[41707]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:18 smithi121 ceph-mon[30020]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:21.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:20 smithi121 ceph-mon[30020]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:21.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:20 smithi121 ceph-mon[41707]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:20 smithi146 ceph-mon[33795]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:22 smithi121 ceph-mon[41707]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:22 smithi121 ceph-mon[30020]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:22 smithi146 ceph-mon[33795]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:24.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:24.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:23 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:23 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:24:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:23 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:24.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:23 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:25.230 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:25.230 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[41707]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:25.230 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:25.230 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:25.230 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:25.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[30020]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:25.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:25.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:25.231 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:24 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:24 smithi146 ceph-mon[33795]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:24 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:24:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:24 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:24:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:24 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:24:27.298 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:26 smithi146 ceph-mon[33795]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:26 smithi121 ceph-mon[41707]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:26 smithi121 ceph-mon[30020]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:27.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:28.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:27 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:27 smithi121 ceph-mon[30020]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:27 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:28.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:27 smithi121 ceph-mon[41707]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:27 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:27 smithi146 ceph-mon[33795]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:30 smithi146 ceph-mon[33795]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:30 smithi121 ceph-mon[41707]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:31.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:30 smithi121 ceph-mon[30020]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:32 smithi146 ceph-mon[33795]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:32 smithi121 ceph-mon[41707]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:32 smithi121 ceph-mon[30020]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:34 smithi146 ceph-mon[33795]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:34 smithi121 ceph-mon[41707]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:34 smithi121 ceph-mon[30020]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:36 smithi146 ceph-mon[33795]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:36 smithi121 ceph-mon[41707]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:36 smithi121 ceph-mon[30020]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:38 smithi146 ceph-mon[33795]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:38 smithi121 ceph-mon[30020]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:38 smithi121 ceph-mon[41707]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:40 smithi146 ceph-mon[33795]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:40 smithi121 ceph-mon[41707]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:40 smithi121 ceph-mon[30020]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:42 smithi146 ceph-mon[33795]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:42 smithi121 ceph-mon[41707]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:43.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:42 smithi121 ceph-mon[30020]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:44 smithi146 ceph-mon[33795]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:44 smithi121 ceph-mon[41707]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:44 smithi121 ceph-mon[30020]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:46 smithi146 ceph-mon[33795]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:46 smithi121 ceph-mon[30020]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:46 smithi121 ceph-mon[41707]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:47.615 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:24:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:48 smithi146 ceph-mon[33795]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:48 smithi121 ceph-mon[41707]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:48 smithi121 ceph-mon[30020]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:50 smithi146 ceph-mon[33795]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:50 smithi121 ceph-mon[30020]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:51.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:50 smithi121 ceph-mon[41707]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:52 smithi146 ceph-mon[33795]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:52 smithi121 ceph-mon[30020]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:52 smithi121 ceph-mon[41707]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:54 smithi146 ceph-mon[33795]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:54 smithi121 ceph-mon[41707]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:54 smithi121 ceph-mon[30020]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:24:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:24:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:24:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:24:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:56 smithi146 ceph-mon[33795]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:56 smithi121 ceph-mon[41707]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:56 smithi121 ceph-mon[30020]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:24:57.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:24:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:24:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:24:58 smithi146 ceph-mon[33795]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:24:58 smithi121 ceph-mon[41707]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:24:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:24:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:24:58 smithi121 ceph-mon[30020]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:00 smithi146 ceph-mon[33795]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:00 smithi121 ceph-mon[41707]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:00 smithi121 ceph-mon[30020]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:02 smithi146 ceph-mon[33795]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:02 smithi121 ceph-mon[41707]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:02 smithi121 ceph-mon[30020]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:04 smithi146 ceph-mon[33795]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:04 smithi121 ceph-mon[41707]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:04 smithi121 ceph-mon[30020]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:06 smithi146 ceph-mon[33795]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:06 smithi121 ceph-mon[41707]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:06 smithi121 ceph-mon[30020]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:07.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:08 smithi146 ceph-mon[33795]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:08 smithi121 ceph-mon[30020]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:08 smithi121 ceph-mon[41707]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:10 smithi146 ceph-mon[33795]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:10 smithi121 ceph-mon[41707]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:10 smithi121 ceph-mon[30020]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:12 smithi146 ceph-mon[33795]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:12 smithi121 ceph-mon[41707]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:12 smithi121 ceph-mon[30020]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:14 smithi146 ceph-mon[33795]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:14 smithi121 ceph-mon[41707]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:14 smithi121 ceph-mon[30020]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:16 smithi121 ceph-mon[41707]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:16 smithi121 ceph-mon[30020]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:16 smithi146 ceph-mon[33795]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:18 smithi121 ceph-mon[41707]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:18 smithi121 ceph-mon[30020]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:18 smithi146 ceph-mon[33795]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:20 smithi121 ceph-mon[41707]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:20 smithi121 ceph-mon[30020]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:20 smithi146 ceph-mon[33795]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:22 smithi121 ceph-mon[41707]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:22 smithi121 ceph-mon[30020]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:22 smithi146 ceph-mon[33795]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:24 smithi121 ceph-mon[41707]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:24 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:24 smithi121 ceph-mon[30020]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:24 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:24 smithi146 ceph-mon[33795]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:24 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:25:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:26.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:26.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:25:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:25 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:25:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:25 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:25:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:25 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:25:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:25 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:25:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:26 smithi121 ceph-mon[41707]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:26 smithi121 ceph-mon[30020]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:26 smithi146 ceph-mon[33795]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:27.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:28 smithi121 ceph-mon[41707]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:28 smithi121 ceph-mon[30020]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:28 smithi146 ceph-mon[33795]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:30 smithi121 ceph-mon[41707]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:30 smithi121 ceph-mon[30020]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:30 smithi146 ceph-mon[33795]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:33.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:32 smithi121 ceph-mon[30020]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:32 smithi121 ceph-mon[41707]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:32 smithi146 ceph-mon[33795]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:34 smithi121 ceph-mon[41707]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:34 smithi121 ceph-mon[30020]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:34 smithi146 ceph-mon[33795]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:36.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:36 smithi146 ceph-mon[33795]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:36 smithi121 ceph-mon[41707]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:36 smithi121 ceph-mon[30020]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:38 smithi121 ceph-mon[41707]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:38 smithi121 ceph-mon[30020]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:38 smithi146 ceph-mon[33795]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:40 smithi121 ceph-mon[41707]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:40 smithi121 ceph-mon[30020]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:40 smithi146 ceph-mon[33795]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:43.080 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:42 smithi121 ceph-mon[41707]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:43.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:42 smithi121 ceph-mon[30020]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:42 smithi146 ceph-mon[33795]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:44 smithi121 ceph-mon[41707]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:44 smithi121 ceph-mon[30020]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:45.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:44 smithi146 ceph-mon[33795]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:46 smithi121 ceph-mon[41707]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:46 smithi121 ceph-mon[30020]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:46 smithi146 ceph-mon[33795]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:47.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:25:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:48 smithi121 ceph-mon[41707]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:48 smithi121 ceph-mon[30020]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:48 smithi146 ceph-mon[33795]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:50 smithi121 ceph-mon[41707]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:50 smithi121 ceph-mon[30020]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:50 smithi146 ceph-mon[33795]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:52 smithi121 ceph-mon[41707]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:52 smithi121 ceph-mon[30020]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:52 smithi146 ceph-mon[33795]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:55.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:54 smithi121 ceph-mon[41707]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:55.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:54 smithi121 ceph-mon[30020]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:54 smithi146 ceph-mon[33795]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:25:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:25:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:25:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:25:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:56 smithi121 ceph-mon[30020]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:57.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:56 smithi121 ceph-mon[41707]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:56 smithi146 ceph-mon[33795]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:25:57.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:25:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:25:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:25:58 smithi121 ceph-mon[41707]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:25:58 smithi121 ceph-mon[30020]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:25:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:25:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:25:58 smithi146 ceph-mon[33795]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:00 smithi121 ceph-mon[30020]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:00 smithi121 ceph-mon[41707]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:00 smithi146 ceph-mon[33795]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:02 smithi121 ceph-mon[30020]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:02 smithi121 ceph-mon[41707]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:02 smithi146 ceph-mon[33795]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:05.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:04 smithi121 ceph-mon[30020]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:04 smithi121 ceph-mon[41707]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:05.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:04 smithi146 ceph-mon[33795]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:06 smithi121 ceph-mon[41707]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:06 smithi121 ceph-mon[30020]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:06 smithi146 ceph-mon[33795]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:07.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:08 smithi121 ceph-mon[41707]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:08 smithi121 ceph-mon[30020]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:08 smithi146 ceph-mon[33795]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:10 smithi121 ceph-mon[41707]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:10 smithi121 ceph-mon[30020]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:10 smithi146 ceph-mon[33795]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:12 smithi121 ceph-mon[41707]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:12 smithi121 ceph-mon[30020]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:12 smithi146 ceph-mon[33795]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:15.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:14 smithi121 ceph-mon[41707]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:15.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:14 smithi121 ceph-mon[30020]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:15.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:14 smithi146 ceph-mon[33795]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:16 smithi121 ceph-mon[41707]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:16 smithi121 ceph-mon[30020]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:16 smithi146 ceph-mon[33795]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:18 smithi121 ceph-mon[30020]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:18 smithi121 ceph-mon[41707]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:18 smithi146 ceph-mon[33795]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:20 smithi121 ceph-mon[41707]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:20 smithi121 ceph-mon[30020]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:20 smithi146 ceph-mon[33795]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:22 smithi121 ceph-mon[41707]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:22 smithi121 ceph-mon[30020]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:22 smithi146 ceph-mon[33795]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:25.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:24 smithi121 ceph-mon[30020]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:25.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:24 smithi121 ceph-mon[41707]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:25.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:24 smithi146 ceph-mon[33795]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:25 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:26.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:25 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:25 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:26:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:26 smithi121 ceph-mon[41707]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:26 smithi121 ceph-mon[30020]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:26 smithi146 ceph-mon[33795]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:27.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:29.056 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:29.056 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:28 smithi121 ceph-mon[41707]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:29.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:29.056 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:28 smithi121 ceph-mon[30020]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:28 smithi146 ceph-mon[33795]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:30.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:30.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:30 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:26:30.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:30 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:26:32.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:32 smithi121 ceph-mon[30020]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:32.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:32.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:32 smithi121 ceph-mon[41707]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:32.831 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:32.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:32 smithi146 ceph-mon[33795]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:34.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:34 smithi121 ceph-mon[41707]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:34.828 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:34 smithi121 ceph-mon[30020]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:34.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:34 smithi146 ceph-mon[33795]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:36.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:36 smithi146 ceph-mon[33795]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:36 smithi121 ceph-mon[30020]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:37.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:36 smithi121 ceph-mon[41707]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:38 smithi146 ceph-mon[33795]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:38 smithi121 ceph-mon[30020]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:38 smithi121 ceph-mon[41707]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:40 smithi146 ceph-mon[33795]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:40 smithi121 ceph-mon[30020]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:41.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:40 smithi121 ceph-mon[41707]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:42 smithi146 ceph-mon[33795]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:42 smithi121 ceph-mon[41707]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:43.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:42 smithi121 ceph-mon[30020]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:44 smithi146 ceph-mon[33795]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:45.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:44 smithi121 ceph-mon[41707]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:44 smithi121 ceph-mon[30020]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:46 smithi146 ceph-mon[33795]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:46 smithi121 ceph-mon[41707]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:46 smithi121 ceph-mon[30020]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:47.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:47.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:48.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:26:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:48 smithi146 ceph-mon[33795]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:48 smithi121 ceph-mon[41707]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:48 smithi121 ceph-mon[30020]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:50 smithi146 ceph-mon[33795]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:50 smithi121 ceph-mon[41707]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:50 smithi121 ceph-mon[30020]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:26:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:52 smithi146 ceph-mon[33795]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:52 smithi121 ceph-mon[41707]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:52 smithi121 ceph-mon[30020]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:54 smithi146 ceph-mon[33795]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:54 smithi121 ceph-mon[41707]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:54 smithi121 ceph-mon[30020]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:26:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:26:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:26:56.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:56 smithi146 ceph-mon[33795]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:56 smithi121 ceph-mon[41707]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:56 smithi121 ceph-mon[30020]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:26:57.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:26:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:26:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:58.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:26:58 smithi146 ceph-mon[33795]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:26:58 smithi121 ceph-mon[41707]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:26:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:26:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:26:58 smithi121 ceph-mon[30020]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:00 smithi146 ceph-mon[33795]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:00 smithi121 ceph-mon[41707]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:00 smithi121 ceph-mon[30020]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:02 smithi146 ceph-mon[33795]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:02 smithi121 ceph-mon[41707]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:02 smithi121 ceph-mon[30020]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:04 smithi146 ceph-mon[33795]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:04 smithi121 ceph-mon[41707]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:04 smithi121 ceph-mon[30020]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:07.041 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:06 smithi121 ceph-mon[41707]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:07.041 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:06 smithi121 ceph-mon[30020]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:06 smithi146 ceph-mon[33795]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:07.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:08 smithi121 ceph-mon[30020]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:08 smithi121 ceph-mon[41707]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:08 smithi146 ceph-mon[33795]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:10 smithi121 ceph-mon[41707]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:10 smithi121 ceph-mon[30020]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:10 smithi146 ceph-mon[33795]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:12 smithi121 ceph-mon[41707]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:12 smithi121 ceph-mon[30020]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:12 smithi146 ceph-mon[33795]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:14 smithi121 ceph-mon[41707]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:15.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:14 smithi121 ceph-mon[30020]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:14 smithi146 ceph-mon[33795]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:16 smithi121 ceph-mon[41707]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:16 smithi121 ceph-mon[30020]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:16 smithi146 ceph-mon[33795]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:17.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:18.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:18.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:18 smithi121 ceph-mon[41707]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:18 smithi121 ceph-mon[30020]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:18 smithi146 ceph-mon[33795]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:20 smithi121 ceph-mon[41707]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:20 smithi121 ceph-mon[30020]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:20 smithi146 ceph-mon[33795]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:22 smithi121 ceph-mon[41707]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:22 smithi121 ceph-mon[30020]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:22 smithi146 ceph-mon[33795]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:24 smithi121 ceph-mon[41707]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:24 smithi121 ceph-mon[30020]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:24 smithi146 ceph-mon[33795]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:26.828 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:26 smithi121 ceph-mon[41707]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:26.829 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:26 smithi121 ceph-mon[30020]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:26 smithi146 ceph-mon[33795]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:27.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:28 smithi121 ceph-mon[41707]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:28 smithi121 ceph-mon[30020]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:28 smithi146 ceph-mon[33795]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:30.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:30 smithi146 ceph-mon[33795]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:30.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:30 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:30 smithi121 ceph-mon[41707]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:30 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:30 smithi121 ceph-mon[30020]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:30 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:27:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:32 smithi146 ceph-mon[33795]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:32.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:32 smithi121 ceph-mon[41707]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:33.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:32 smithi121 ceph-mon[30020]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:35.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:34 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:27:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:34 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:27:37.225 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:36 smithi146 ceph-mon[33795]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:36 smithi121 ceph-mon[41707]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:36 smithi121 ceph-mon[30020]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:27:37.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:38 smithi121 ceph-mon[41707]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:38 smithi121 ceph-mon[30020]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:39.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:39.419 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:38 smithi146 ceph-mon[33795]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:27:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:40 smithi121 ceph-mon[41707]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:40 smithi121 ceph-mon[30020]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:40 smithi146 ceph-mon[33795]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:27:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:42 smithi121 ceph-mon[41707]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:42 smithi121 ceph-mon[30020]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:42 smithi146 ceph-mon[33795]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:44 smithi121 ceph-mon[41707]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:44 smithi121 ceph-mon[30020]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:44 smithi146 ceph-mon[33795]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:46 smithi121 ceph-mon[41707]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:46 smithi121 ceph-mon[30020]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:46 smithi146 ceph-mon[33795]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 778 MiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 53 op/s 2024-02-10T06:27:47.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:48.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:48.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:27:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:48 smithi121 ceph-mon[41707]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:48 smithi121 ceph-mon[30020]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:48 smithi146 ceph-mon[33795]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:50 smithi121 ceph-mon[41707]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:51.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:50 smithi121 ceph-mon[30020]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:51.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:50 smithi146 ceph-mon[33795]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:52 smithi121 ceph-mon[30020]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:53.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:52 smithi121 ceph-mon[41707]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:53.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:52 smithi146 ceph-mon[33795]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-10T06:27:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:54 smithi121 ceph-mon[41707]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:54 smithi121 ceph-mon[30020]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:55.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:27:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:27:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:27:55.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:54 smithi146 ceph-mon[33795]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:56 smithi121 ceph-mon[41707]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:56 smithi121 ceph-mon[30020]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:56 smithi146 ceph-mon[33795]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-10T06:27:57.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:27:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:27:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:57 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:27:57 smithi121 ceph-mon[41707]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-10T06:27:58.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:57 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:27:57 smithi121 ceph-mon[30020]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-10T06:27:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:57 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:27:58.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:27:57 smithi146 ceph-mon[33795]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-10T06:28:00.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:00 smithi146 ceph-mon[33795]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:00 smithi121 ceph-mon[41707]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:00 smithi121 ceph-mon[30020]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:02 smithi146 ceph-mon[33795]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:02.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:02 smithi121 ceph-mon[41707]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:02 smithi121 ceph-mon[30020]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:04.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:04 smithi146 ceph-mon[33795]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:04 smithi121 ceph-mon[41707]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:04 smithi121 ceph-mon[30020]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:06.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:06 smithi146 ceph-mon[33795]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:06 smithi121 ceph-mon[41707]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:07.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:06 smithi121 ceph-mon[30020]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:07.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:08 smithi146 ceph-mon[33795]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:08 smithi121 ceph-mon[41707]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:08 smithi121 ceph-mon[30020]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:10.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:10 smithi146 ceph-mon[33795]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:10 smithi121 ceph-mon[41707]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:10 smithi121 ceph-mon[30020]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:12.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:12 smithi146 ceph-mon[33795]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:12 smithi121 ceph-mon[41707]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:13.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:12 smithi121 ceph-mon[30020]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:14.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:14 smithi146 ceph-mon[33795]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:14 smithi121 ceph-mon[41707]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:14 smithi121 ceph-mon[30020]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:16.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:16 smithi146 ceph-mon[33795]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:16 smithi121 ceph-mon[41707]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:16 smithi121 ceph-mon[30020]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:17.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:18.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:18 smithi146 ceph-mon[33795]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:18 smithi121 ceph-mon[30020]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:18 smithi121 ceph-mon[41707]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:20 smithi121 ceph-mon[41707]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:20 smithi121 ceph-mon[30020]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:20 smithi146 ceph-mon[33795]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:22 smithi121 ceph-mon[41707]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:22 smithi121 ceph-mon[30020]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:22 smithi146 ceph-mon[33795]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:24 smithi121 ceph-mon[41707]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:24 smithi121 ceph-mon[30020]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:24 smithi146 ceph-mon[33795]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:26 smithi121 ceph-mon[41707]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:26 smithi121 ceph-mon[30020]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:26 smithi146 ceph-mon[33795]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:27.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:28 smithi121 ceph-mon[30020]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:28 smithi121 ceph-mon[41707]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:28 smithi146 ceph-mon[33795]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:31.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:30 smithi121 ceph-mon[41707]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:31.080 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:30 smithi121 ceph-mon[30020]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:30 smithi146 ceph-mon[33795]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:32 smithi121 ceph-mon[41707]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:32 smithi121 ceph-mon[30020]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:32 smithi146 ceph-mon[33795]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:34 smithi121 ceph-mon[41707]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:34 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:34 smithi121 ceph-mon[30020]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:34 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:34 smithi146 ceph-mon[33795]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:34 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:28:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:36.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[41707]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:36 smithi121 ceph-mon[30020]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:28:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:28:36.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:28:36.673 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:36 smithi146 ceph-mon[33795]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:37.851 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:38 smithi146 ceph-mon[33795]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:38 smithi121 ceph-mon[30020]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:38 smithi121 ceph-mon[41707]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:40.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:40 smithi146 ceph-mon[33795]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:40 smithi121 ceph-mon[41707]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:41.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:40 smithi121 ceph-mon[30020]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:42.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:42 smithi146 ceph-mon[33795]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:42 smithi121 ceph-mon[41707]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:43.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:42 smithi121 ceph-mon[30020]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:44.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:44 smithi146 ceph-mon[33795]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:45.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:44 smithi121 ceph-mon[41707]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:45.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:44 smithi121 ceph-mon[30020]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:46.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:46 smithi146 ceph-mon[33795]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:46 smithi121 ceph-mon[41707]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:47.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:46 smithi121 ceph-mon[30020]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:47.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:28:48.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:48.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:48 smithi146 ceph-mon[33795]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:48 smithi121 ceph-mon[41707]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:48 smithi121 ceph-mon[30020]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:50.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:50 smithi146 ceph-mon[33795]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:50 smithi121 ceph-mon[41707]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:50 smithi121 ceph-mon[30020]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:52.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:52 smithi146 ceph-mon[33795]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:53.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:52 smithi121 ceph-mon[41707]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:52 smithi121 ceph-mon[30020]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:54.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:54 smithi146 ceph-mon[33795]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:55.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:54 smithi121 ceph-mon[41707]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:55.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:54 smithi121 ceph-mon[30020]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:28:55.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:28:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:28:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:28:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:56 smithi121 ceph-mon[41707]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:57.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:56 smithi121 ceph-mon[30020]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:56 smithi146 ceph-mon[33795]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:28:57.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:28:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:28:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:28:58 smithi121 ceph-mon[41707]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:28:58 smithi121 ceph-mon[30020]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:28:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:28:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:28:58 smithi146 ceph-mon[33795]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:00 smithi121 ceph-mon[30020]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:00 smithi121 ceph-mon[41707]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:00 smithi146 ceph-mon[33795]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:02 smithi121 ceph-mon[41707]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:02 smithi121 ceph-mon[30020]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:02 smithi146 ceph-mon[33795]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:05.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:04 smithi121 ceph-mon[41707]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:05.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:04 smithi121 ceph-mon[30020]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:04 smithi146 ceph-mon[33795]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:05.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:06 smithi121 ceph-mon[41707]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:06 smithi121 ceph-mon[30020]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:06 smithi146 ceph-mon[33795]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:07.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:08 smithi121 ceph-mon[41707]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:08 smithi121 ceph-mon[30020]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:08 smithi146 ceph-mon[33795]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:10 smithi121 ceph-mon[41707]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:10 smithi121 ceph-mon[30020]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:10 smithi146 ceph-mon[33795]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:12 smithi121 ceph-mon[41707]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:12 smithi121 ceph-mon[30020]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:12 smithi146 ceph-mon[33795]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:13.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:29:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a[67197]: logger=cleanup t=2024-02-10T06:29:13.594099485Z level=info msg="Completed cleanup jobs" duration=3.325434ms 2024-02-10T06:29:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:14 smithi121 ceph-mon[41707]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:14 smithi121 ceph-mon[30020]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:14 smithi146 ceph-mon[33795]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:17.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:16 smithi121 ceph-mon[30020]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:17.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:16 smithi121 ceph-mon[41707]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:16 smithi146 ceph-mon[33795]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:17.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:17.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:18.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:18.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:19.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:18 smithi121 ceph-mon[41707]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:18 smithi121 ceph-mon[30020]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:18 smithi146 ceph-mon[33795]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:21.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:20 smithi121 ceph-mon[41707]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:20 smithi121 ceph-mon[30020]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:20 smithi146 ceph-mon[33795]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:22 smithi121 ceph-mon[41707]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:22 smithi121 ceph-mon[30020]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:22 smithi146 ceph-mon[33795]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:25.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:24 smithi121 ceph-mon[41707]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:25.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:24 smithi121 ceph-mon[30020]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:24 smithi146 ceph-mon[33795]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:26 smithi121 ceph-mon[41707]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:26 smithi121 ceph-mon[30020]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:26 smithi146 ceph-mon[33795]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:27.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:28 smithi121 ceph-mon[41707]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:28 smithi121 ceph-mon[30020]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:28 smithi146 ceph-mon[33795]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:30 smithi121 ceph-mon[41707]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:30 smithi121 ceph-mon[30020]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:30 smithi146 ceph-mon[33795]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:32 smithi121 ceph-mon[41707]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:32 smithi121 ceph-mon[30020]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:32 smithi146 ceph-mon[33795]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:35.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:34 smithi121 ceph-mon[41707]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:35.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:34 smithi121 ceph-mon[30020]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:34 smithi146 ceph-mon[33795]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:36.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:35 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:36.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:35 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:36.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:35 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:29:37.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:37.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.073 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:37.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:37.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:36 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:29:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:36 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:29:37.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:39.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:38 smithi121 ceph-mon[41707]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:39.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:38 smithi121 ceph-mon[30020]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:38 smithi146 ceph-mon[33795]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:40 smithi121 ceph-mon[41707]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:40 smithi121 ceph-mon[30020]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:40 smithi146 ceph-mon[33795]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:43.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:42 smithi121 ceph-mon[41707]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:43.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:42 smithi121 ceph-mon[30020]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:42 smithi146 ceph-mon[33795]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:45.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:45.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:44 smithi121 ceph-mon[41707]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:45.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:44 smithi121 ceph-mon[30020]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:44 smithi146 ceph-mon[33795]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:47.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:46 smithi121 ceph-mon[41707]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:47.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:46 smithi121 ceph-mon[30020]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:46 smithi146 ceph-mon[33795]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:47.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:47.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:48.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:48.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:29:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:48 smithi121 ceph-mon[41707]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:48 smithi121 ceph-mon[30020]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:48 smithi146 ceph-mon[33795]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:50 smithi121 ceph-mon[41707]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:51.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:50 smithi121 ceph-mon[30020]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:50 smithi146 ceph-mon[33795]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:53.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:52 smithi121 ceph-mon[30020]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:52 smithi121 ceph-mon[41707]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:52 smithi146 ceph-mon[33795]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:54 smithi121 ceph-mon[41707]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:55.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:54 smithi121 ceph-mon[30020]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:55.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:29:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:29:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:29:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:54 smithi146 ceph-mon[33795]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:29:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:56 smithi121 ceph-mon[41707]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:57.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:56 smithi121 ceph-mon[30020]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:56 smithi146 ceph-mon[33795]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:29:57.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:29:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:29:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:29:58 smithi121 ceph-mon[41707]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:29:58 smithi121 ceph-mon[30020]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:29:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:29:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:29:58 smithi146 ceph-mon[33795]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:00 smithi121 ceph-mon[30020]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:01.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:00 smithi121 ceph-mon[30020]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:30:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:00 smithi121 ceph-mon[41707]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:01.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:00 smithi121 ceph-mon[41707]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:30:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:00 smithi146 ceph-mon[33795]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:00 smithi146 ceph-mon[33795]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-02-10T06:30:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:02 smithi121 ceph-mon[41707]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:03.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:02 smithi121 ceph-mon[30020]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:02 smithi146 ceph-mon[33795]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:05.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:04 smithi121 ceph-mon[41707]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:04 smithi121 ceph-mon[30020]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:05.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:04 smithi146 ceph-mon[33795]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:30:07.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:06 smithi121 ceph-mon[41707]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:06 smithi121 ceph-mon[30020]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:06 smithi146 ceph-mon[33795]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:30:07.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:09.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:08 smithi121 ceph-mon[30020]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:09.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:08 smithi121 ceph-mon[41707]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:08 smithi146 ceph-mon[33795]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:10.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:09 smithi121 ceph-mon[30020]: osdmap e754: 8 total, 8 up, 8 in 2024-02-10T06:30:10.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:09 smithi121 ceph-mon[41707]: osdmap e754: 8 total, 8 up, 8 in 2024-02-10T06:30:10.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:09 smithi146 ceph-mon[33795]: osdmap e754: 8 total, 8 up, 8 in 2024-02-10T06:30:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:10 smithi146 ceph-mon[33795]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:10 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:10 smithi146 ceph-mon[33795]: osdmap e755: 8 total, 8 up, 8 in 2024-02-10T06:30:11.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[30020]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:11.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[30020]: osdmap e755: 8 total, 8 up, 8 in 2024-02-10T06:30:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[41707]: pgmap v1640: 196 pgs: 196 active+clean; 582 KiB data, 795 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:10 smithi121 ceph-mon[41707]: osdmap e755: 8 total, 8 up, 8 in 2024-02-10T06:30:12.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:11 smithi146 ceph-mon[33795]: osdmap e756: 8 total, 8 up, 8 in 2024-02-10T06:30:12.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:11 smithi121 ceph-mon[30020]: osdmap e756: 8 total, 8 up, 8 in 2024-02-10T06:30:12.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:11 smithi121 ceph-mon[41707]: osdmap e756: 8 total, 8 up, 8 in 2024-02-10T06:30:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:12 smithi146 ceph-mon[33795]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:30:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:12 smithi146 ceph-mon[33795]: osdmap e757: 8 total, 8 up, 8 in 2024-02-10T06:30:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:12 smithi121 ceph-mon[30020]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:30:13.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:12 smithi121 ceph-mon[30020]: osdmap e757: 8 total, 8 up, 8 in 2024-02-10T06:30:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:12 smithi121 ceph-mon[41707]: pgmap v1643: 196 pgs: 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T06:30:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:12 smithi121 ceph-mon[41707]: osdmap e757: 8 total, 8 up, 8 in 2024-02-10T06:30:14.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:13 smithi146 ceph-mon[33795]: osdmap e758: 8 total, 8 up, 8 in 2024-02-10T06:30:14.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:13 smithi121 ceph-mon[41707]: osdmap e758: 8 total, 8 up, 8 in 2024-02-10T06:30:14.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:13 smithi121 ceph-mon[30020]: osdmap e758: 8 total, 8 up, 8 in 2024-02-10T06:30:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:14 smithi146 ceph-mon[33795]: pgmap v1646: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:14 smithi146 ceph-mon[33795]: osdmap e759: 8 total, 8 up, 8 in 2024-02-10T06:30:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:14 smithi121 ceph-mon[41707]: pgmap v1646: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:14 smithi121 ceph-mon[41707]: osdmap e759: 8 total, 8 up, 8 in 2024-02-10T06:30:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:14 smithi121 ceph-mon[30020]: pgmap v1646: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:14 smithi121 ceph-mon[30020]: osdmap e759: 8 total, 8 up, 8 in 2024-02-10T06:30:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:15.898 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Running main() from gmock_main.cc 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [----------] Global test environment set-up. 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: handle_notify cookie 94790497348880 notify_id 3234110373888 notifier_gid 24818 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1808736 ms) 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Trying... 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: handle_notify cookie 94790510074160 notify_id 3246995275781 notifier_gid 42134 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Waiting for 3.000000000s 2024-02-10T06:30:15.899 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Timed out. 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Flushing... 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: Flushed... 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7070 ms) 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1815806 ms total) 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [----------] Global test environment tear-down 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1815806 ms total) 2024-02-10T06:30:15.900 INFO:tasks.workunit.client.0.smithi121.stdout: watch_notify: [ PASSED ] 2 tests. 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81776 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81776 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82171 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82171 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82368 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82368 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.901 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82251 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82251 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82552 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82552 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82128 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82128 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81675 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81675 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82718 2024-02-10T06:30:15.902 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82718 2024-02-10T06:30:16.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:15 smithi146 ceph-mon[33795]: osdmap e760: 8 total, 8 up, 8 in 2024-02-10T06:30:16.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:15 smithi121 ceph-mon[30020]: osdmap e760: 8 total, 8 up, 8 in 2024-02-10T06:30:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:15 smithi121 ceph-mon[41707]: osdmap e760: 8 total, 8 up, 8 in 2024-02-10T06:30:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:16 smithi146 ceph-mon[33795]: pgmap v1649: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:16 smithi146 ceph-mon[33795]: osdmap e761: 8 total, 8 up, 8 in 2024-02-10T06:30:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:16 smithi121 ceph-mon[30020]: pgmap v1649: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:16 smithi121 ceph-mon[30020]: osdmap e761: 8 total, 8 up, 8 in 2024-02-10T06:30:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:16 smithi121 ceph-mon[41707]: pgmap v1649: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 800 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:17.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:16 smithi121 ceph-mon[41707]: osdmap e761: 8 total, 8 up, 8 in 2024-02-10T06:30:17.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:17 smithi146 ceph-mon[33795]: osdmap e762: 8 total, 8 up, 8 in 2024-02-10T06:30:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:17 smithi121 ceph-mon[41707]: osdmap e762: 8 total, 8 up, 8 in 2024-02-10T06:30:18.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:17 smithi121 ceph-mon[30020]: osdmap e762: 8 total, 8 up, 8 in 2024-02-10T06:30:18.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:18 smithi146 ceph-mon[33795]: pgmap v1652: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:18 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:18 smithi146 ceph-mon[33795]: osdmap e763: 8 total, 8 up, 8 in 2024-02-10T06:30:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[41707]: pgmap v1652: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[41707]: osdmap e763: 8 total, 8 up, 8 in 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[30020]: pgmap v1652: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:18 smithi121 ceph-mon[30020]: osdmap e763: 8 total, 8 up, 8 in 2024-02-10T06:30:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:19 smithi146 ceph-mon[33795]: osdmap e764: 8 total, 8 up, 8 in 2024-02-10T06:30:20.332 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:19 smithi121 ceph-mon[41707]: osdmap e764: 8 total, 8 up, 8 in 2024-02-10T06:30:20.332 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:19 smithi121 ceph-mon[30020]: osdmap e764: 8 total, 8 up, 8 in 2024-02-10T06:30:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:20 smithi121 ceph-mon[41707]: pgmap v1655: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:20 smithi121 ceph-mon[41707]: osdmap e765: 8 total, 8 up, 8 in 2024-02-10T06:30:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:20 smithi121 ceph-mon[30020]: pgmap v1655: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:21.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:20 smithi121 ceph-mon[30020]: osdmap e765: 8 total, 8 up, 8 in 2024-02-10T06:30:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:20 smithi146 ceph-mon[33795]: pgmap v1655: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:20 smithi146 ceph-mon[33795]: osdmap e765: 8 total, 8 up, 8 in 2024-02-10T06:30:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:21 smithi121 ceph-mon[41707]: osdmap e766: 8 total, 8 up, 8 in 2024-02-10T06:30:22.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:21 smithi121 ceph-mon[41707]: osdmap e767: 8 total, 8 up, 8 in 2024-02-10T06:30:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:21 smithi121 ceph-mon[30020]: osdmap e766: 8 total, 8 up, 8 in 2024-02-10T06:30:22.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:21 smithi121 ceph-mon[30020]: osdmap e767: 8 total, 8 up, 8 in 2024-02-10T06:30:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:21 smithi146 ceph-mon[33795]: osdmap e766: 8 total, 8 up, 8 in 2024-02-10T06:30:22.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:21 smithi146 ceph-mon[33795]: osdmap e767: 8 total, 8 up, 8 in 2024-02-10T06:30:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:22 smithi121 ceph-mon[41707]: pgmap v1658: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:22 smithi121 ceph-mon[41707]: osdmap e768: 8 total, 8 up, 8 in 2024-02-10T06:30:23.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:22 smithi121 ceph-mon[30020]: pgmap v1658: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:22 smithi121 ceph-mon[30020]: osdmap e768: 8 total, 8 up, 8 in 2024-02-10T06:30:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:22 smithi146 ceph-mon[33795]: pgmap v1658: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:23.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:22 smithi146 ceph-mon[33795]: osdmap e768: 8 total, 8 up, 8 in 2024-02-10T06:30:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:24 smithi121 ceph-mon[41707]: pgmap v1661: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:25.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:24 smithi121 ceph-mon[41707]: osdmap e769: 8 total, 8 up, 8 in 2024-02-10T06:30:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:24 smithi121 ceph-mon[30020]: pgmap v1661: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:24 smithi121 ceph-mon[30020]: osdmap e769: 8 total, 8 up, 8 in 2024-02-10T06:30:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:24 smithi146 ceph-mon[33795]: pgmap v1661: 196 pgs: 17 creating+peering, 15 unknown, 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:25.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:24 smithi146 ceph-mon[33795]: osdmap e769: 8 total, 8 up, 8 in 2024-02-10T06:30:26.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:25 smithi121 ceph-mon[30020]: osdmap e770: 8 total, 8 up, 8 in 2024-02-10T06:30:26.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:25 smithi121 ceph-mon[41707]: osdmap e770: 8 total, 8 up, 8 in 2024-02-10T06:30:26.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:25 smithi146 ceph-mon[33795]: osdmap e770: 8 total, 8 up, 8 in 2024-02-10T06:30:27.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[41707]: pgmap v1664: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:27.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:27.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[41707]: osdmap e771: 8 total, 8 up, 8 in 2024-02-10T06:30:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[30020]: pgmap v1664: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:27.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:26 smithi121 ceph-mon[30020]: osdmap e771: 8 total, 8 up, 8 in 2024-02-10T06:30:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:26 smithi146 ceph-mon[33795]: pgmap v1664: 164 pgs: 164 active+clean; 582 KiB data, 806 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:26 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:27.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:26 smithi146 ceph-mon[33795]: osdmap e771: 8 total, 8 up, 8 in 2024-02-10T06:30:27.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:27 smithi121 ceph-mon[41707]: osdmap e772: 8 total, 8 up, 8 in 2024-02-10T06:30:28.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:27 smithi121 ceph-mon[41707]: osdmap e773: 8 total, 8 up, 8 in 2024-02-10T06:30:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:27 smithi121 ceph-mon[30020]: osdmap e772: 8 total, 8 up, 8 in 2024-02-10T06:30:28.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:27 smithi121 ceph-mon[30020]: osdmap e773: 8 total, 8 up, 8 in 2024-02-10T06:30:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:27 smithi146 ceph-mon[33795]: osdmap e772: 8 total, 8 up, 8 in 2024-02-10T06:30:28.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:27 smithi146 ceph-mon[33795]: osdmap e773: 8 total, 8 up, 8 in 2024-02-10T06:30:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:29.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[41707]: pgmap v1667: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:29.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[41707]: osdmap e774: 8 total, 8 up, 8 in 2024-02-10T06:30:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[30020]: pgmap v1667: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:28 smithi121 ceph-mon[30020]: osdmap e774: 8 total, 8 up, 8 in 2024-02-10T06:30:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:28 smithi146 ceph-mon[33795]: pgmap v1667: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:29.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:28 smithi146 ceph-mon[33795]: osdmap e774: 8 total, 8 up, 8 in 2024-02-10T06:30:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:30 smithi121 ceph-mon[41707]: pgmap v1670: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:30 smithi121 ceph-mon[41707]: osdmap e775: 8 total, 8 up, 8 in 2024-02-10T06:30:31.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:30 smithi121 ceph-mon[30020]: pgmap v1670: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:31.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:30 smithi121 ceph-mon[30020]: osdmap e775: 8 total, 8 up, 8 in 2024-02-10T06:30:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:30 smithi146 ceph-mon[33795]: pgmap v1670: 196 pgs: 1 creating+peering, 31 unknown, 164 active+clean; 582 KiB data, 815 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:30:31.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:30 smithi146 ceph-mon[33795]: osdmap e775: 8 total, 8 up, 8 in 2024-02-10T06:30:32.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:31 smithi121 ceph-mon[41707]: osdmap e776: 8 total, 8 up, 8 in 2024-02-10T06:30:32.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:31 smithi121 ceph-mon[30020]: osdmap e776: 8 total, 8 up, 8 in 2024-02-10T06:30:32.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:31 smithi146 ceph-mon[33795]: osdmap e776: 8 total, 8 up, 8 in 2024-02-10T06:30:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[30020]: pgmap v1673: 196 pgs: 196 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:30:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[30020]: osdmap e777: 8 total, 8 up, 8 in 2024-02-10T06:30:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[41707]: pgmap v1673: 196 pgs: 196 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:30:33.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[41707]: osdmap e777: 8 total, 8 up, 8 in 2024-02-10T06:30:33.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:32 smithi146 ceph-mon[33795]: pgmap v1673: 196 pgs: 196 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-10T06:30:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:32 smithi146 ceph-mon[33795]: osdmap e777: 8 total, 8 up, 8 in 2024-02-10T06:30:33.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:34 smithi121 ceph-mon[41707]: osdmap e778: 8 total, 8 up, 8 in 2024-02-10T06:30:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:34 smithi121 ceph-mon[30020]: osdmap e778: 8 total, 8 up, 8 in 2024-02-10T06:30:34.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:34 smithi146 ceph-mon[33795]: osdmap e778: 8 total, 8 up, 8 in 2024-02-10T06:30:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[41707]: pgmap v1676: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[41707]: osdmap e779: 8 total, 8 up, 8 in 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[30020]: pgmap v1676: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[30020]: osdmap e779: 8 total, 8 up, 8 in 2024-02-10T06:30:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:35 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:35 smithi146 ceph-mon[33795]: pgmap v1676: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:35 smithi146 ceph-mon[33795]: osdmap e779: 8 total, 8 up, 8 in 2024-02-10T06:30:35.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:35 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:36 smithi121 ceph-mon[41707]: osdmap e780: 8 total, 8 up, 8 in 2024-02-10T06:30:36.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:36 smithi121 ceph-mon[41707]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:36 smithi121 ceph-mon[30020]: osdmap e780: 8 total, 8 up, 8 in 2024-02-10T06:30:36.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:36 smithi121 ceph-mon[30020]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:36 smithi146 ceph-mon[33795]: osdmap e780: 8 total, 8 up, 8 in 2024-02-10T06:30:36.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:36 smithi146 ceph-mon[33795]: pgmap v1679: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 837 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:37.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:37 smithi146 ceph-mon[33795]: osdmap e781: 8 total, 8 up, 8 in 2024-02-10T06:30:37.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:37 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:37 smithi121 ceph-mon[41707]: osdmap e781: 8 total, 8 up, 8 in 2024-02-10T06:30:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:37 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:37 smithi121 ceph-mon[30020]: osdmap e781: 8 total, 8 up, 8 in 2024-02-10T06:30:37.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:37 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:30:37.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: osdmap e782: 8 total, 8 up, 8 in 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[41707]: pgmap v1682: 164 pgs: 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: osdmap e782: 8 total, 8 up, 8 in 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:30:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:38 smithi121 ceph-mon[30020]: pgmap v1682: 164 pgs: 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: osdmap e782: 8 total, 8 up, 8 in 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:38.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:38 smithi146 ceph-mon[33795]: pgmap v1682: 164 pgs: 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:39 smithi121 ceph-mon[41707]: osdmap e783: 8 total, 8 up, 8 in 2024-02-10T06:30:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:39 smithi121 ceph-mon[30020]: osdmap e783: 8 total, 8 up, 8 in 2024-02-10T06:30:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:39 smithi146 ceph-mon[33795]: osdmap e783: 8 total, 8 up, 8 in 2024-02-10T06:30:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[41707]: osdmap e784: 8 total, 8 up, 8 in 2024-02-10T06:30:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[41707]: pgmap v1685: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[30020]: osdmap e784: 8 total, 8 up, 8 in 2024-02-10T06:30:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[30020]: pgmap v1685: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:40 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:40 smithi146 ceph-mon[33795]: osdmap e784: 8 total, 8 up, 8 in 2024-02-10T06:30:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:40 smithi146 ceph-mon[33795]: pgmap v1685: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 842 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:40.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:40 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:41 smithi121 ceph-mon[41707]: osdmap e785: 8 total, 8 up, 8 in 2024-02-10T06:30:41.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:41 smithi121 ceph-mon[30020]: osdmap e785: 8 total, 8 up, 8 in 2024-02-10T06:30:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:41 smithi146 ceph-mon[33795]: osdmap e785: 8 total, 8 up, 8 in 2024-02-10T06:30:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[41707]: osdmap e786: 8 total, 8 up, 8 in 2024-02-10T06:30:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[41707]: pgmap v1688: 196 pgs: 196 active+clean; 583 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:30:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[41707]: osdmap e787: 8 total, 8 up, 8 in 2024-02-10T06:30:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[30020]: osdmap e786: 8 total, 8 up, 8 in 2024-02-10T06:30:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[30020]: pgmap v1688: 196 pgs: 196 active+clean; 583 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:30:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:42 smithi121 ceph-mon[30020]: osdmap e787: 8 total, 8 up, 8 in 2024-02-10T06:30:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:42 smithi146 ceph-mon[33795]: osdmap e786: 8 total, 8 up, 8 in 2024-02-10T06:30:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:42 smithi146 ceph-mon[33795]: pgmap v1688: 196 pgs: 196 active+clean; 583 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-10T06:30:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:42 smithi146 ceph-mon[33795]: osdmap e787: 8 total, 8 up, 8 in 2024-02-10T06:30:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:44 smithi121 ceph-mon[41707]: osdmap e788: 8 total, 8 up, 8 in 2024-02-10T06:30:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:44 smithi121 ceph-mon[41707]: pgmap v1691: 164 pgs: 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:44 smithi121 ceph-mon[30020]: osdmap e788: 8 total, 8 up, 8 in 2024-02-10T06:30:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:44 smithi121 ceph-mon[30020]: pgmap v1691: 164 pgs: 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:44 smithi146 ceph-mon[33795]: osdmap e788: 8 total, 8 up, 8 in 2024-02-10T06:30:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:44 smithi146 ceph-mon[33795]: pgmap v1691: 164 pgs: 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:45 smithi121 ceph-mon[30020]: osdmap e789: 8 total, 8 up, 8 in 2024-02-10T06:30:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:45 smithi121 ceph-mon[41707]: osdmap e789: 8 total, 8 up, 8 in 2024-02-10T06:30:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:45 smithi146 ceph-mon[33795]: osdmap e789: 8 total, 8 up, 8 in 2024-02-10T06:30:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:46 smithi146 ceph-mon[33795]: osdmap e790: 8 total, 8 up, 8 in 2024-02-10T06:30:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:46 smithi146 ceph-mon[33795]: pgmap v1694: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:46 smithi121 ceph-mon[41707]: osdmap e790: 8 total, 8 up, 8 in 2024-02-10T06:30:46.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:46 smithi121 ceph-mon[41707]: pgmap v1694: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:46 smithi121 ceph-mon[30020]: osdmap e790: 8 total, 8 up, 8 in 2024-02-10T06:30:46.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:46 smithi121 ceph-mon[30020]: pgmap v1694: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 860 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:47 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:47 smithi146 ceph-mon[33795]: osdmap e791: 8 total, 8 up, 8 in 2024-02-10T06:30:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:47 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:47.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:47 smithi121 ceph-mon[41707]: osdmap e791: 8 total, 8 up, 8 in 2024-02-10T06:30:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:47 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:47.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:47 smithi121 ceph-mon[30020]: osdmap e791: 8 total, 8 up, 8 in 2024-02-10T06:30:47.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:48 smithi146 ceph-mon[33795]: osdmap e792: 8 total, 8 up, 8 in 2024-02-10T06:30:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:48 smithi146 ceph-mon[33795]: pgmap v1697: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:48.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:48 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[41707]: osdmap e792: 8 total, 8 up, 8 in 2024-02-10T06:30:48.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[41707]: pgmap v1697: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[30020]: osdmap e792: 8 total, 8 up, 8 in 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[30020]: pgmap v1697: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:48.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:48 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:30:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:49 smithi146 ceph-mon[33795]: osdmap e793: 8 total, 8 up, 8 in 2024-02-10T06:30:49.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:49 smithi121 ceph-mon[30020]: osdmap e793: 8 total, 8 up, 8 in 2024-02-10T06:30:49.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:49 smithi121 ceph-mon[41707]: osdmap e793: 8 total, 8 up, 8 in 2024-02-10T06:30:50.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:50 smithi146 ceph-mon[33795]: osdmap e794: 8 total, 8 up, 8 in 2024-02-10T06:30:50.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:50 smithi146 ceph-mon[33795]: pgmap v1700: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:50.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:50 smithi121 ceph-mon[41707]: osdmap e794: 8 total, 8 up, 8 in 2024-02-10T06:30:50.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:50 smithi121 ceph-mon[41707]: pgmap v1700: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:50 smithi121 ceph-mon[30020]: osdmap e794: 8 total, 8 up, 8 in 2024-02-10T06:30:50.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:50 smithi121 ceph-mon[30020]: pgmap v1700: 196 pgs: 196 active+clean; 583 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-10T06:30:51.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:51 smithi146 ceph-mon[33795]: osdmap e795: 8 total, 8 up, 8 in 2024-02-10T06:30:51.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:51 smithi121 ceph-mon[41707]: osdmap e795: 8 total, 8 up, 8 in 2024-02-10T06:30:51.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:51 smithi121 ceph-mon[30020]: osdmap e795: 8 total, 8 up, 8 in 2024-02-10T06:30:52.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:52 smithi146 ceph-mon[33795]: osdmap e796: 8 total, 8 up, 8 in 2024-02-10T06:30:52.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:52 smithi146 ceph-mon[33795]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:52 smithi121 ceph-mon[41707]: osdmap e796: 8 total, 8 up, 8 in 2024-02-10T06:30:52.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:52 smithi121 ceph-mon[41707]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:52 smithi121 ceph-mon[30020]: osdmap e796: 8 total, 8 up, 8 in 2024-02-10T06:30:52.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:52 smithi121 ceph-mon[30020]: pgmap v1703: 164 pgs: 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:53.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:53 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:53.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:53 smithi146 ceph-mon[33795]: osdmap e797: 8 total, 8 up, 8 in 2024-02-10T06:30:53.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:53 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:53.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:53 smithi121 ceph-mon[41707]: osdmap e797: 8 total, 8 up, 8 in 2024-02-10T06:30:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:53 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:53.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:53 smithi121 ceph-mon[30020]: osdmap e797: 8 total, 8 up, 8 in 2024-02-10T06:30:54.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:54 smithi146 ceph-mon[33795]: osdmap e798: 8 total, 8 up, 8 in 2024-02-10T06:30:54.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:54 smithi146 ceph-mon[33795]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:54.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:54 smithi121 ceph-mon[41707]: osdmap e798: 8 total, 8 up, 8 in 2024-02-10T06:30:54.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:54 smithi121 ceph-mon[41707]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:54.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:54 smithi121 ceph-mon[30020]: osdmap e798: 8 total, 8 up, 8 in 2024-02-10T06:30:54.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:54 smithi121 ceph-mon[30020]: pgmap v1706: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-10T06:30:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:55 smithi121 ceph-mon[41707]: osdmap e799: 8 total, 8 up, 8 in 2024-02-10T06:30:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:55 smithi121 ceph-mon[30020]: osdmap e799: 8 total, 8 up, 8 in 2024-02-10T06:30:55.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:30:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:30:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:30:55.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:55 smithi146 ceph-mon[33795]: osdmap e799: 8 total, 8 up, 8 in 2024-02-10T06:30:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:56 smithi146 ceph-mon[33795]: osdmap e800: 8 total, 8 up, 8 in 2024-02-10T06:30:56.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:56 smithi146 ceph-mon[33795]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:56.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:56 smithi121 ceph-mon[30020]: osdmap e800: 8 total, 8 up, 8 in 2024-02-10T06:30:56.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:56 smithi121 ceph-mon[30020]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:56.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:56 smithi121 ceph-mon[41707]: osdmap e800: 8 total, 8 up, 8 in 2024-02-10T06:30:56.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:56 smithi121 ceph-mon[41707]: pgmap v1709: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 845 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:30:57.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:57 smithi121 ceph-mon[30020]: osdmap e801: 8 total, 8 up, 8 in 2024-02-10T06:30:57.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:57 smithi121 ceph-mon[41707]: osdmap e801: 8 total, 8 up, 8 in 2024-02-10T06:30:57.607 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:57 smithi146 ceph-mon[33795]: osdmap e801: 8 total, 8 up, 8 in 2024-02-10T06:30:57.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:30:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[41707]: osdmap e802: 8 total, 8 up, 8 in 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[41707]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[30020]: osdmap e802: 8 total, 8 up, 8 in 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:58 smithi121 ceph-mon[30020]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:58 smithi146 ceph-mon[33795]: osdmap e802: 8 total, 8 up, 8 in 2024-02-10T06:30:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:30:58.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:58 smithi146 ceph-mon[33795]: pgmap v1712: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:30:59.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:59 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:59.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:30:59 smithi121 ceph-mon[41707]: osdmap e803: 8 total, 8 up, 8 in 2024-02-10T06:30:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:59 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:59.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:30:59 smithi121 ceph-mon[30020]: osdmap e803: 8 total, 8 up, 8 in 2024-02-10T06:30:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:59 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:30:59.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:30:59 smithi146 ceph-mon[33795]: osdmap e803: 8 total, 8 up, 8 in 2024-02-10T06:31:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:00 smithi121 ceph-mon[41707]: osdmap e804: 8 total, 8 up, 8 in 2024-02-10T06:31:00.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:00 smithi121 ceph-mon[41707]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:00.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:00 smithi121 ceph-mon[30020]: osdmap e804: 8 total, 8 up, 8 in 2024-02-10T06:31:00.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:00 smithi121 ceph-mon[30020]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:00 smithi146 ceph-mon[33795]: osdmap e804: 8 total, 8 up, 8 in 2024-02-10T06:31:00.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:00 smithi146 ceph-mon[33795]: pgmap v1715: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:01.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:01 smithi121 ceph-mon[41707]: osdmap e805: 8 total, 8 up, 8 in 2024-02-10T06:31:01.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:01 smithi121 ceph-mon[30020]: osdmap e805: 8 total, 8 up, 8 in 2024-02-10T06:31:01.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:01 smithi146 ceph-mon[33795]: osdmap e805: 8 total, 8 up, 8 in 2024-02-10T06:31:02.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:02 smithi121 ceph-mon[41707]: osdmap e806: 8 total, 8 up, 8 in 2024-02-10T06:31:02.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:02 smithi121 ceph-mon[41707]: pgmap v1718: 164 pgs: 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:02.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:02 smithi121 ceph-mon[30020]: osdmap e806: 8 total, 8 up, 8 in 2024-02-10T06:31:02.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:02 smithi121 ceph-mon[30020]: pgmap v1718: 164 pgs: 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:02.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:02 smithi146 ceph-mon[33795]: osdmap e806: 8 total, 8 up, 8 in 2024-02-10T06:31:02.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:02 smithi146 ceph-mon[33795]: pgmap v1718: 164 pgs: 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:03.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:03 smithi121 ceph-mon[30020]: osdmap e807: 8 total, 8 up, 8 in 2024-02-10T06:31:03.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:03 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:03 smithi121 ceph-mon[41707]: osdmap e807: 8 total, 8 up, 8 in 2024-02-10T06:31:03.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:03 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:03 smithi146 ceph-mon[33795]: osdmap e807: 8 total, 8 up, 8 in 2024-02-10T06:31:03.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:03 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:04.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:04 smithi121 ceph-mon[41707]: osdmap e808: 8 total, 8 up, 8 in 2024-02-10T06:31:04.579 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:04 smithi121 ceph-mon[41707]: pgmap v1721: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:04.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:04 smithi121 ceph-mon[30020]: osdmap e808: 8 total, 8 up, 8 in 2024-02-10T06:31:04.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:04 smithi121 ceph-mon[30020]: pgmap v1721: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:04 smithi146 ceph-mon[33795]: osdmap e808: 8 total, 8 up, 8 in 2024-02-10T06:31:04.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:04 smithi146 ceph-mon[33795]: pgmap v1721: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:05.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:05 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:05 smithi121 ceph-mon[30020]: osdmap e809: 8 total, 8 up, 8 in 2024-02-10T06:31:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:05 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:05.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:05 smithi121 ceph-mon[41707]: osdmap e809: 8 total, 8 up, 8 in 2024-02-10T06:31:05.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:05 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:05.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:05 smithi146 ceph-mon[33795]: osdmap e809: 8 total, 8 up, 8 in 2024-02-10T06:31:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:06 smithi121 ceph-mon[41707]: osdmap e810: 8 total, 8 up, 8 in 2024-02-10T06:31:06.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:06 smithi121 ceph-mon[41707]: pgmap v1724: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:06 smithi121 ceph-mon[30020]: osdmap e810: 8 total, 8 up, 8 in 2024-02-10T06:31:06.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:06 smithi121 ceph-mon[30020]: pgmap v1724: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:06 smithi146 ceph-mon[33795]: osdmap e810: 8 total, 8 up, 8 in 2024-02-10T06:31:06.672 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:06 smithi146 ceph-mon[33795]: pgmap v1724: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:07.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:08 smithi146 ceph-mon[33795]: pgmap v1725: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 947 B/s rd, 189 B/s wr, 1 op/s 2024-02-10T06:31:08.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:08 smithi121 ceph-mon[41707]: pgmap v1725: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 947 B/s rd, 189 B/s wr, 1 op/s 2024-02-10T06:31:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:08 smithi121 ceph-mon[30020]: pgmap v1725: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 947 B/s rd, 189 B/s wr, 1 op/s 2024-02-10T06:31:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:10 smithi121 ceph-mon[41707]: pgmap v1726: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 799 B/s rd, 159 B/s wr, 0 op/s 2024-02-10T06:31:11.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:10 smithi121 ceph-mon[41707]: osdmap e811: 8 total, 8 up, 8 in 2024-02-10T06:31:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:10 smithi121 ceph-mon[30020]: pgmap v1726: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 799 B/s rd, 159 B/s wr, 0 op/s 2024-02-10T06:31:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:10 smithi121 ceph-mon[30020]: osdmap e811: 8 total, 8 up, 8 in 2024-02-10T06:31:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:10 smithi146 ceph-mon[33795]: pgmap v1726: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 799 B/s rd, 159 B/s wr, 0 op/s 2024-02-10T06:31:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:10 smithi146 ceph-mon[33795]: osdmap e811: 8 total, 8 up, 8 in 2024-02-10T06:31:13.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:12 smithi121 ceph-mon[41707]: pgmap v1728: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 B/s wr, 1 op/s 2024-02-10T06:31:13.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:12 smithi121 ceph-mon[30020]: pgmap v1728: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 B/s wr, 1 op/s 2024-02-10T06:31:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:12 smithi146 ceph-mon[33795]: pgmap v1728: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 138 B/s wr, 1 op/s 2024-02-10T06:31:15.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:14 smithi121 ceph-mon[41707]: pgmap v1729: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 B/s wr, 1 op/s 2024-02-10T06:31:15.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:14 smithi121 ceph-mon[30020]: pgmap v1729: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 B/s wr, 1 op/s 2024-02-10T06:31:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:14 smithi146 ceph-mon[33795]: pgmap v1729: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 B/s wr, 1 op/s 2024-02-10T06:31:15.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:17.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:16 smithi121 ceph-mon[41707]: pgmap v1730: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:31:17.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:16 smithi121 ceph-mon[30020]: pgmap v1730: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:31:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:16 smithi146 ceph-mon[33795]: pgmap v1730: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T06:31:17.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[30020]: pgmap v1731: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[41707]: pgmap v1731: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:18 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:18 smithi146 ceph-mon[33795]: pgmap v1731: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:18 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:20.710 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: Running main() from gmock_main.cc 2024-02-10T06:31:20.710 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-02-10T06:31:20.710 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] Global test environment set-up. 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (1810754 ms) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5064 ms) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4024 ms) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5035 ms) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7024 ms) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (1831902 ms total) 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-02-10T06:31:20.711 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-02-10T06:31:20.712 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5047 ms) 2024-02-10T06:31:20.712 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-02-10T06:31:20.712 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6033 ms) 2024-02-10T06:31:20.712 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8059 ms) 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6047 ms) 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4037 ms) 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi121-82837-11. 2024-02-10T06:31:20.715 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: Waiting for snaps to purge. 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19497 ms) 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48721 ms total) 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [----------] Global test environment tear-down 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (1880623 ms total) 2024-02-10T06:31:20.716 INFO:tasks.workunit.client.0.smithi121.stdout: snapshots: [ PASSED ] 11 tests. 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82036 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82036 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81453 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81453 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81512 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81512 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81975 2024-02-10T06:31:20.717 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81975 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=81559 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 81559 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82078 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82078 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82092 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82092 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82338 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82338 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ for t in "${!pids[@]}" 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=82820 2024-02-10T06:31:20.718 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 82820 2024-02-10T06:31:20.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ exit 0 2024-02-10T06:31:20.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ cleanup 2024-02-10T06:31:20.719 INFO:tasks.workunit.client.0.smithi121.stderr:+ pkill -P 81448 2024-02-10T06:31:20.796 INFO:tasks.workunit.client.0.smithi121.stderr:+ true 2024-02-10T06:31:20.796 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-10T06:31:20.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-10T06:31:20.830 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-02-10T06:31:20.831 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-02-10T06:31:20.831 DEBUG:teuthology.orchestra.run.smithi121:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-02-10T06:31:20.890 INFO:tasks.workunit.client.0.smithi121.stderr:++ uuidgen 2024-02-10T06:31:20.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ p=263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:31:20.903 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool create 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 12 2024-02-10T06:31:20.992 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 -- 172.21.15.121:0/1587039585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a810b820 msgr2=0x7fd7a810dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/1587039585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a810b820 0x7fd7a810dc90 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009ca0 tx=0x7fd7a002c180 comp rx=0 tx=0).stop 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 -- 172.21.15.121:0/1587039585 shutdown_connections 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/1587039585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a810b820 0x7fd7a810dc90 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/1587039585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a8102d70 0x7fd7a810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/1587039585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102450 0x7fd7a8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 -- 172.21.15.121:0/1587039585 >> 172.21.15.121:0/1587039585 conn(0x7fd7a80772b0 msgr2=0x7fd7a80776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 -- 172.21.15.121:0/1587039585 shutdown_connections 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.989+0000 7fd7ae89a640 1 -- 172.21.15.121:0/1587039585 wait complete. 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 Processor -- start 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 -- start start 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 0x7fd7a8197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:20.993 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a810b820 0x7fd7a819b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a819bcf0 con 0x7fd7a8102450 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a819be60 con 0x7fd7a8102d70 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ae89a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a819c020 con 0x7fd7a810b820 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58238/0 (socket says 172.21.15.121:58238) 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 -- 172.21.15.121:0/391207558 learned_addr learned my addr 172.21.15.121:0/391207558 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7ace37640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a810b820 0x7fd7a819b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a77fe640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 0x7fd7a8197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 -- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a810b820 msgr2=0x7fd7a819b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a810b820 0x7fd7a819b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 -- 172.21.15.121:0/391207558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 msgr2=0x7fd7a8197370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 0x7fd7a8197370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a7fff640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c009490 con 0x7fd7a8102450 2024-02-10T06:31:20.994 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.990+0000 7fd7a77fe640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 0x7fd7a8197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7a7fff640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd79800b4a0 tx=0x7fd79800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798010610 con 0x7fd7a8102450 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd798010bf0 con 0x7fd7a8102450 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a0009900 con 0x7fd7a8102450 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798017bc0 con 0x7fd7a8102450 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.991+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a81a39a0 con 0x7fd7a8102450 2024-02-10T06:31:20.995 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.992+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a8068560 con 0x7fd7a8102450 2024-02-10T06:31:20.996 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.992+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fd798017d40 con 0x7fd7a8102450 2024-02-10T06:31:20.996 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.993+0000 7fd7a57fa640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 0x7fd79006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:20.996 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.993+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(812..812 src has 253..812) v4 ==== 8687+0+0 (secure 0 0 0) 0x7fd79808d420 con 0x7fd7a8102450 2024-02-10T06:31:20.996 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.993+0000 7fd7a77fe640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 0x7fd79006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:20.996 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.993+0000 7fd7a77fe640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 0x7fd79006fef0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8077b40 tx=0x7fd79c005f90 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:20.998 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:20.998+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd798056400 con 0x7fd7a8102450 2024-02-10T06:31:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:20 smithi121 ceph-mon[41707]: pgmap v1732: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:21.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:20 smithi121 ceph-mon[30020]: pgmap v1732: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:20 smithi146 ceph-mon[33795]: pgmap v1732: 196 pgs: 196 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:21.173 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.172+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12} v 0) v1 -- 0x7fd7a8106020 con 0x7fd7a8102450 2024-02-10T06:31:21.702 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.701+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]=0 pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' created v813) v1 ==== 176+0+0 (secure 0 0 0) 0x7fd798059ac0 con 0x7fd7a8102450 2024-02-10T06:31:21.812 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.811+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12} v 0) v1 -- 0x7fd7a8110820 con 0x7fd7a8102450 2024-02-10T06:31:21.813 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.812+0000 7fd7a57fa640 1 -- 172.21.15.121:0/391207558 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]=0 pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' already exists v813) v1 ==== 183+0+0 (secure 0 0 0) 0x7fd7980140c0 con 0x7fd7a8102450 2024-02-10T06:31:21.813 INFO:tasks.workunit.client.0.smithi121.stderr:pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' already exists 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 msgr2=0x7fd79006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 0x7fd79006fef0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8077b40 tx=0x7fd79c005f90 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 msgr2=0x7fd7a8196e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fd79800b4a0 tx=0x7fd79800b960 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 shutdown_connections 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd79006da30 0x7fd79006fef0 unknown :-1 s=CLOSED pgs=3541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7a810b820 0x7fd7a819b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a8102d70 0x7fd7a8197370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 --2- 172.21.15.121:0/391207558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7a8102450 0x7fd7a8196e30 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 >> 172.21.15.121:0/391207558 conn(0x7fd7a80772b0 msgr2=0x7fd7a81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 shutdown_connections 2024-02-10T06:31:21.815 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.814+0000 7fd7ae89a640 1 -- 172.21.15.121:0/391207558 wait complete. 2024-02-10T06:31:21.824 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_objects 10 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- 172.21.15.121:0/1862113815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 msgr2=0x7f7d9c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 --2- 172.21.15.121:0/1862113815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c100740 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009900 tx=0x7f7d9802bc30 comp rx=0 tx=0).stop 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- 172.21.15.121:0/1862113815 shutdown_connections 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 --2- 172.21.15.121:0/1862113815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 0x7f7d9c10d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 --2- 172.21.15.121:0/1862113815 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c100d10 0x7f7d9c104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 --2- 172.21.15.121:0/1862113815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c100740 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.910 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- 172.21.15.121:0/1862113815 >> 172.21.15.121:0/1862113815 conn(0x7f7d9c077250 msgr2=0x7f7d9c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- 172.21.15.121:0/1862113815 shutdown_connections 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- 172.21.15.121:0/1862113815 wait complete. 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 Processor -- start 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.905+0000 7f7da368f640 1 -- start start 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 0x7f7d9c196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c19a690 0x7f7d9c19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19b080 con 0x7f7d9c105430 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19b1f0 con 0x7f7d9c19a690 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9c19b360 con 0x7f7d9c100360 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37300/0 (socket says 172.21.15.121:37300) 2024-02-10T06:31:21.911 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da0c2a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 0x7f7d9c196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 -- 172.21.15.121:0/3464461427 learned_addr learned my addr 172.21.15.121:0/3464461427 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da1c2c640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c19a690 0x7f7d9c19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 -- 172.21.15.121:0/3464461427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c19a690 msgr2=0x7f7d9c19ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c19a690 0x7f7d9c19ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 -- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 msgr2=0x7f7d9c196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 0x7f7d9c196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d94009840 con 0x7f7d9c100360 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da142b640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005940 tx=0x7f7d980042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d980044e0 con 0x7f7d9c100360 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7d98003c80 con 0x7f7d9c100360 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98005540 con 0x7f7d9c100360 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d98009560 con 0x7f7d9c100360 2024-02-10T06:31:21.912 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.906+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9c1a2c30 con 0x7f7d9c100360 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.907+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c068110 con 0x7f7d9c100360 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.908+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f7d98004d90 con 0x7f7d9c100360 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.908+0000 7f7d927fc640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 0x7f7d8806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.908+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(813..813 src has 253..813) v4 ==== 9062+0+0 (secure 0 0 0) 0x7f7d980b0330 con 0x7f7d9c100360 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.908+0000 7f7da0c2a640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 0x7f7d8806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.908+0000 7f7da0c2a640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 0x7f7d8806fef0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c1068d0 tx=0x7f7d8c00b390 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:21.913 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:21.913+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d980790e0 con 0x7f7d9c100360 2024-02-10T06:31:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:21 smithi121 ceph-mon[41707]: osdmap e812: 8 total, 8 up, 8 in 2024-02-10T06:31:22.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:21 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:22.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:21 smithi121 ceph-mon[30020]: osdmap e812: 8 total, 8 up, 8 in 2024-02-10T06:31:22.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:21 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:22.088 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:22.087+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f7d9c100d10 con 0x7f7d9c100360 2024-02-10T06:31:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:21 smithi146 ceph-mon[33795]: osdmap e812: 8 total, 8 up, 8 in 2024-02-10T06:31:22.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:21 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:22.713 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:22.712+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v814) v1 ==== 223+0+0 (secure 0 0 0) 0x7f7d9807c7a0 con 0x7f7d9c100360 2024-02-10T06:31:22.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:22.822+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f7d9c10f6e0 con 0x7f7d9c100360 2024-02-10T06:31:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: pgmap v1734: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]': finished 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: osdmap e813: 8 total, 8 up, 8 in 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: pgmap v1734: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]': finished 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: osdmap e813: 8 total, 8 up, 8 in 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:22 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: pgmap v1734: 164 pgs: 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]': finished 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: osdmap e813: 8 total, 8 up, 8 in 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/391207558' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pg_num": 12}]: dispatch 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:22 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:23.719 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.718+0000 7f7d927fc640 1 -- 172.21.15.121:0/3464461427 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v815) v1 ==== 223+0+0 (secure 0 0 0) 0x7f7d9807c980 con 0x7f7d9c100360 2024-02-10T06:31:23.720 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_objects = 10 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:31:23.722 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 msgr2=0x7f7d8806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:23.722 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 0x7f7d8806fef0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c1068d0 tx=0x7f7d8c00b390 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 msgr2=0x7f7d9c195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005940 tx=0x7f7d980042f0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 shutdown_connections 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f7d8806da30 0x7f7d8806fef0 unknown :-1 s=CLOSED pgs=3542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d9c19a690 0x7f7d9c19ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9c105430 0x7f7d9c196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 --2- 172.21.15.121:0/3464461427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d9c100360 0x7f7d9c195dd0 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 >> 172.21.15.121:0/3464461427 conn(0x7f7d9c077250 msgr2=0x7f7d9c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 shutdown_connections 2024-02-10T06:31:23.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.721+0000 7f7da368f640 1 -- 172.21.15.121:0/3464461427 wait complete. 2024-02-10T06:31:23.733 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool application enable 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 rados 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/252370324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8101e10 msgr2=0x7fb2f8105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/252370324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8101e10 0x7fb2f8105e50 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009900 tx=0x7fb2f402bc30 comp rx=0 tx=0).stop 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/252370324 shutdown_connections 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/252370324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8106530 0x7fb2f810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/252370324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8101e10 0x7fb2f8105e50 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/252370324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f8101460 0x7fb2f8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.823 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/252370324 >> 172.21.15.121:0/252370324 conn(0x7fb2f80fccc0 msgr2=0x7fb2f80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/252370324 shutdown_connections 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/252370324 wait complete. 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 Processor -- start 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.820+0000 7fb2ffbdd640 1 -- start start 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8101460 0x7fb2f8196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 0x7fb2f819bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f819c100 con 0x7fb2f819b710 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f819c270 con 0x7fb2f8101460 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f819c3e0 con 0x7fb2f8106530 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fe17a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 0x7fb2f819bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:23.824 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:37320/0 (socket says 172.21.15.121:37320) 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 -- 172.21.15.121:0/2529815266 learned_addr learned my addr 172.21.15.121:0/2529815266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd979640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8101460 0x7fb2f8196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 -- 172.21.15.121:0/2529815266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8101460 msgr2=0x7fb2f8196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8101460 0x7fb2f8196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 -- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 msgr2=0x7fb2f819bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 0x7fb2f819bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8009490 con 0x7fb2f8106530 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fe17a640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 0x7fb2f819bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2fd178640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4000c00 tx=0x7fb2f4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:23.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40035f0 con 0x7fb2f8106530 2024-02-10T06:31:23.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb2f4003bd0 con 0x7fb2f8106530 2024-02-10T06:31:23.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f4009560 con 0x7fb2f8106530 2024-02-10T06:31:23.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f40054c0 con 0x7fb2f8106530 2024-02-10T06:31:23.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.821+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f81a3cb0 con 0x7fb2f8106530 2024-02-10T06:31:23.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.822+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f8068110 con 0x7fb2f8106530 2024-02-10T06:31:23.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.823+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fb2f4003750 con 0x7fb2f8106530 2024-02-10T06:31:23.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.823+0000 7fb2eeffd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 0x7fb2e406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:23.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.823+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(815..815 src has 253..815) v4 ==== 9062+0+0 (secure 0 0 0) 0x7fb2f40b0530 con 0x7fb2f8106530 2024-02-10T06:31:23.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.823+0000 7fb2fd979640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 0x7fb2e406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:23.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.824+0000 7fb2fd979640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 0x7fb2e406fef0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009cf0 tx=0x7fb2e8005f90 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:23.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:23.828+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb2f40792e0 con 0x7fb2f8106530 2024-02-10T06:31:24.008 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:24.005+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"} v 0) v1 -- 0x7fb2f8101e10 con 0x7fb2f8106530 2024-02-10T06:31:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:24.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[41707]: osdmap e814: 8 total, 8 up, 8 in 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[30020]: osdmap e814: 8 total, 8 up, 8 in 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:23 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:23 smithi146 ceph-mon[33795]: osdmap e814: 8 total, 8 up, 8 in 2024-02-10T06:31:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:23 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3464461427' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:23 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:31:24.719 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:24.718+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]=0 enabled application 'rados' on pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' v816) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb2f407c9a0 con 0x7fb2f8106530 2024-02-10T06:31:24.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:24.825+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"} v 0) v1 -- 0x7fb2f8110840 con 0x7fb2f8106530 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[41707]: pgmap v1737: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[41707]: osdmap e815: 8 total, 8 up, 8 in 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[30020]: pgmap v1737: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[30020]: osdmap e815: 8 total, 8 up, 8 in 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.010 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:24 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:24 smithi146 ceph-mon[33795]: pgmap v1737: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:31:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:24 smithi146 ceph-mon[33795]: osdmap e815: 8 total, 8 up, 8 in 2024-02-10T06:31:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:24 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:24 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:25.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:25.734 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.733+0000 7fb2eeffd640 1 -- 172.21.15.121:0/2529815266 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]=0 enabled application 'rados' on pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' v817) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb2f4043020 con 0x7fb2f8106530 2024-02-10T06:31:25.734 INFO:tasks.workunit.client.0.smithi121.stderr:enabled application 'rados' on pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' 2024-02-10T06:31:25.737 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 msgr2=0x7fb2e406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:25.737 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 0x7fb2e406fef0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009cf0 tx=0x7fb2e8005f90 comp rx=0 tx=0).stop 2024-02-10T06:31:25.737 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 msgr2=0x7fb2f8197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:25.737 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4000c00 tx=0x7fb2f4004270 comp rx=0 tx=0).stop 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 shutdown_connections 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb2e406da30 0x7fb2e406fef0 unknown :-1 s=CLOSED pgs=3543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2f819b710 0x7fb2f819bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2f8106530 0x7fb2f8197390 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 --2- 172.21.15.121:0/2529815266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f8101460 0x7fb2f8196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 >> 172.21.15.121:0/2529815266 conn(0x7fb2f80fccc0 msgr2=0x7fb2f80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 shutdown_connections 2024-02-10T06:31:25.738 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:25.736+0000 7fb2ffbdd640 1 -- 172.21.15.121:0/2529815266 wait complete. 2024-02-10T06:31:25.748 INFO:tasks.workunit.client.0.smithi121.stderr:++ seq 1 10 2024-02-10T06:31:25.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.754 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj1 /etc/passwd 2024-02-10T06:31:25.793 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.793 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj2 /etc/passwd 2024-02-10T06:31:25.829 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.829 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj3 /etc/passwd 2024-02-10T06:31:25.864 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.864 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj4 /etc/passwd 2024-02-10T06:31:25.899 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.899 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj5 /etc/passwd 2024-02-10T06:31:25.933 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.933 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj6 /etc/passwd 2024-02-10T06:31:25.969 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:25.969 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj7 /etc/passwd 2024-02-10T06:31:26.003 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:26.003 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj8 /etc/passwd 2024-02-10T06:31:26.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:26.037 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj9 /etc/passwd 2024-02-10T06:31:26.072 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:31:26.072 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put obj10 /etc/passwd 2024-02-10T06:31:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[41707]: osdmap e816: 8 total, 8 up, 8 in 2024-02-10T06:31:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:26.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[30020]: osdmap e816: 8 total, 8 up, 8 in 2024-02-10T06:31:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:26.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:25 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:26.106 INFO:tasks.workunit.client.0.smithi121.stderr:+ sleep 30 2024-02-10T06:31:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:25 smithi146 ceph-mon[33795]: osdmap e816: 8 total, 8 up, 8 in 2024-02-10T06:31:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:25 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2529815266' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:26.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:25 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]: dispatch 2024-02-10T06:31:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[30020]: pgmap v1740: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:27.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[30020]: osdmap e817: 8 total, 8 up, 8 in 2024-02-10T06:31:27.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[41707]: pgmap v1740: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:27.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:27.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:26 smithi121 ceph-mon[41707]: osdmap e817: 8 total, 8 up, 8 in 2024-02-10T06:31:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:26 smithi146 ceph-mon[33795]: pgmap v1740: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 854 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:31:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:26 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "app": "rados"}]': finished 2024-02-10T06:31:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:26 smithi146 ceph-mon[33795]: osdmap e817: 8 total, 8 up, 8 in 2024-02-10T06:31:27.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:28 smithi121 ceph-mon[30020]: pgmap v1742: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:29.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:28 smithi121 ceph-mon[41707]: pgmap v1742: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:28 smithi146 ceph-mon[33795]: pgmap v1742: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:30 smithi121 ceph-mon[41707]: pgmap v1743: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 12 KiB/s wr, 1 op/s 2024-02-10T06:31:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:30 smithi121 ceph-mon[30020]: pgmap v1743: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 12 KiB/s wr, 1 op/s 2024-02-10T06:31:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:30 smithi146 ceph-mon[33795]: pgmap v1743: 176 pgs: 176 active+clean; 651 KiB data, 858 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 12 KiB/s wr, 1 op/s 2024-02-10T06:31:33.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:32 smithi121 ceph-mon[41707]: pgmap v1744: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:33.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:32 smithi121 ceph-mon[30020]: pgmap v1744: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:32 smithi146 ceph-mon[33795]: pgmap v1744: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-02-10T06:31:34.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:33 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:34.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:33 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:33 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[41707]: pgmap v1745: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[41707]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_objects: 10) 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[41707]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[41707]: osdmap e818: 8 total, 8 up, 8 in 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[30020]: pgmap v1745: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[30020]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_objects: 10) 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[30020]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:31:35.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:34 smithi121 ceph-mon[30020]: osdmap e818: 8 total, 8 up, 8 in 2024-02-10T06:31:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:34 smithi146 ceph-mon[33795]: pgmap v1745: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-02-10T06:31:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:34 smithi146 ceph-mon[33795]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_objects: 10) 2024-02-10T06:31:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:34 smithi146 ceph-mon[33795]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:31:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:34 smithi146 ceph-mon[33795]: osdmap e818: 8 total, 8 up, 8 in 2024-02-10T06:31:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:36 smithi121 ceph-mon[41707]: pgmap v1747: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-02-10T06:31:37.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:36 smithi121 ceph-mon[30020]: pgmap v1747: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-02-10T06:31:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:36 smithi146 ceph-mon[33795]: pgmap v1747: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-02-10T06:31:37.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:37 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:37.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:38.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:37 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:38.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:37 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:31:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:38 smithi146 ceph-mon[33795]: pgmap v1748: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:38.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:38 smithi121 ceph-mon[30020]: pgmap v1748: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:39.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:38 smithi121 ceph-mon[41707]: pgmap v1748: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:39.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[41707]: pgmap v1749: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:41.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[30020]: pgmap v1749: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:41.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:40 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:40 smithi146 ceph-mon[33795]: pgmap v1749: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-02-10T06:31:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:40 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:40 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: pgmap v1750: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: pgmap v1750: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:43.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:43.330 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:43.331 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:42 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: pgmap v1750: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:31:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:42 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:45.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:44 smithi121 ceph-mon[30020]: pgmap v1751: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:44 smithi121 ceph-mon[41707]: pgmap v1751: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:45.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:44 smithi146 ceph-mon[33795]: pgmap v1751: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:47.333 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:46 smithi121 ceph-mon[41707]: pgmap v1752: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:31:47.333 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:46 smithi121 ceph-mon[30020]: pgmap v1752: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:31:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:46 smithi146 ceph-mon[33795]: pgmap v1752: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:31:47.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:49.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[41707]: pgmap v1753: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[30020]: pgmap v1753: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:49.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:48 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:48 smithi146 ceph-mon[33795]: pgmap v1753: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:48 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:31:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:48 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:31:51.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:50 smithi121 ceph-mon[41707]: pgmap v1754: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:51.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:50 smithi121 ceph-mon[30020]: pgmap v1754: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:50 smithi146 ceph-mon[33795]: pgmap v1754: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:53.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:52 smithi121 ceph-mon[41707]: pgmap v1755: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:53.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:52 smithi121 ceph-mon[30020]: pgmap v1755: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:52 smithi146 ceph-mon[33795]: pgmap v1755: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:31:55.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:54 smithi121 ceph-mon[41707]: pgmap v1756: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:55.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:54 smithi121 ceph-mon[30020]: pgmap v1756: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:55.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:31:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:31:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:31:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:54 smithi146 ceph-mon[33795]: pgmap v1756: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:56.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=158620 2024-02-10T06:31:56.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_objects 100 2024-02-10T06:31:56.108 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put onemore /etc/passwd 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 -- 172.21.15.121:0/4049588986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c100080 msgr2=0x7f283c10e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 --2- 172.21.15.121:0/4049588986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c100080 0x7f283c10e0b0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f282c009890 tx=0x7f282c02bb70 comp rx=0 tx=0).stop 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 -- 172.21.15.121:0/4049588986 shutdown_connections 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 --2- 172.21.15.121:0/4049588986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c100080 0x7f283c10e0b0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 --2- 172.21.15.121:0/4049588986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c0ff6c0 0x7f283c0ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:56.199 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 --2- 172.21.15.121:0/4049588986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c106200 0x7f283c1065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 -- 172.21.15.121:0/4049588986 >> 172.21.15.121:0/4049588986 conn(0x7f283c076640 msgr2=0x7f283c0fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 -- 172.21.15.121:0/4049588986 shutdown_connections 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.192+0000 7f2841556640 1 -- 172.21.15.121:0/4049588986 wait complete. 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.193+0000 7f2841556640 1 Processor -- start 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.193+0000 7f2841556640 1 -- start start 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c100080 0x7f283c1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:56.200 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 0x7f283c19b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c19bc60 con 0x7f283c0ff6c0 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c19bdd0 con 0x7f283c100080 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c19bf90 con 0x7f283c106200 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48094/0 (socket says 172.21.15.121:48094) 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 -- 172.21.15.121:0/3143550386 learned_addr learned my addr 172.21.15.121:0/3143550386 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283b7fe640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 0x7f283c19b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:56.201 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283a7fc640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c100080 0x7f283c1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 -- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 msgr2=0x7f283c19b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 0x7f283c19b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 -- 172.21.15.121:0/3143550386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c100080 msgr2=0x7f283c1972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c100080 0x7f283c1972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834009560 con 0x7f283c0ff6c0 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283b7fe640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 0x7f283c19b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:31:56.202 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f283affd640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f283400e9c0 tx=0x7f283400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283400cde0 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282c009490 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f28340044a0 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834010430 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.194+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c1a3910 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.195+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c068560 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.196+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f2834003ee0 con 0x7f283c0ff6c0 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.196+0000 7f2823fff640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 0x7f282406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:56.203 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.196+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(818..818 src has 253..818) v4 ==== 9075+0+0 (secure 0 0 0) 0x7f2834014070 con 0x7f283c0ff6c0 2024-02-10T06:31:56.204 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.196+0000 7f283a7fc640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 0x7f282406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:56.204 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.196+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=819}) v3 -- 0x7f28240794c0 con 0x7f283c0ff6c0 2024-02-10T06:31:56.204 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.197+0000 7f283a7fc640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 0x7f282406fef0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7f283c105510 tx=0x7f28300075b0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:56.204 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.201+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2834057160 con 0x7f283c0ff6c0 2024-02-10T06:31:56.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.377+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f283c110890 con 0x7f283c0ff6c0 2024-02-10T06:31:56.780 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.779+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v819) v1 ==== 225+0+0 (secure 0 0 0) 0x7f283405a820 con 0x7f283c0ff6c0 2024-02-10T06:31:56.782 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.781+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 8 ==== osd_map(819..819 src has 253..819) v4 ==== 628+0+0 (secure 0 0 0) 0x7f283405ef60 con 0x7f283c0ff6c0 2024-02-10T06:31:56.782 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.781+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=820}) v3 -- 0x7f282407a460 con 0x7f283c0ff6c0 2024-02-10T06:31:56.887 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:56.886+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f283c1980f0 con 0x7f283c0ff6c0 2024-02-10T06:31:57.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:56 smithi121 ceph-mon[41707]: pgmap v1757: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:57.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:56 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:57.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:56 smithi121 ceph-mon[30020]: pgmap v1757: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:57.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:56 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:56 smithi146 ceph-mon[33795]: pgmap v1757: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:31:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:56 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:57.786 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.784+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v820) v1 ==== 225+0+0 (secure 0 0 0) 0x7f283405aa00 con 0x7f283c0ff6c0 2024-02-10T06:31:57.786 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_objects = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:31:57.786 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.784+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 <== mon.0 v2:172.21.15.121:3300/0 10 ==== osd_map(820..820 src has 253..820) v4 ==== 628+0+0 (secure 0 0 0) 0x7f2834063520 con 0x7f283c0ff6c0 2024-02-10T06:31:57.786 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.784+0000 7f2823fff640 1 -- 172.21.15.121:0/3143550386 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=821}) v3 -- 0x7f282407a550 con 0x7f283c0ff6c0 2024-02-10T06:31:57.789 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.788+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 msgr2=0x7f282406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:57.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.788+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 0x7f282406fef0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7f283c105510 tx=0x7f28300075b0 comp rx=0 tx=0).stop 2024-02-10T06:31:57.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.788+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 msgr2=0x7f283c196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:57.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.788+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f283400e9c0 tx=0x7f283400ee80 comp rx=0 tx=0).stop 2024-02-10T06:31:57.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 shutdown_connections 2024-02-10T06:31:57.790 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f282406da30 0x7f282406fef0 unknown :-1 s=CLOSED pgs=3555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:57.791 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f283c106200 0x7f283c19b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:57.791 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c100080 0x7f283c1972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:57.791 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 --2- 172.21.15.121:0/3143550386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f283c0ff6c0 0x7f283c196da0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:57.791 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.789+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 >> 172.21.15.121:0/3143550386 conn(0x7f283c076640 msgr2=0x7f283c103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:57.792 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.790+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 shutdown_connections 2024-02-10T06:31:57.792 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:57.790+0000 7f2841556640 1 -- 172.21.15.121:0/3143550386 wait complete. 2024-02-10T06:31:57.802 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 158620 2024-02-10T06:31:57.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:31:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:31:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:57 smithi146 ceph-mon[33795]: osdmap e819: 8 total, 8 up, 8 in 2024-02-10T06:31:57.922 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:57 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:58.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[30020]: osdmap e819: 8 total, 8 up, 8 in 2024-02-10T06:31:58.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:58.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:58.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[41707]: osdmap e819: 8 total, 8 up, 8 in 2024-02-10T06:31:58.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:57 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]: dispatch 2024-02-10T06:31:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[41707]: pgmap v1759: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:59.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[41707]: osdmap e820: 8 total, 8 up, 8 in 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[30020]: pgmap v1759: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:59.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:58 smithi121 ceph-mon[30020]: osdmap e820: 8 total, 8 up, 8 in 2024-02-10T06:31:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:58 smithi146 ceph-mon[33795]: pgmap v1759: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:31:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:31:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:58 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3143550386' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "100"}]': finished 2024-02-10T06:31:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:58 smithi146 ceph-mon[33795]: osdmap e820: 8 total, 8 up, 8 in 2024-02-10T06:31:59.700 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 0 -ne 0 ']' 2024-02-10T06:31:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+ true 2024-02-10T06:31:59.701 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put twomore /etc/passwd 2024-02-10T06:31:59.738 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_bytes 100 2024-02-10T06:31:59.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/4191832411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc105430 msgr2=0x7fd8cc10d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:59.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/4191832411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc105430 0x7fd8cc10d000 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009be0 tx=0x7fd8c402bef0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.827 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/4191832411 shutdown_connections 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/4191832411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc105430 0x7fd8cc10d000 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/4191832411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100d10 0x7fd8cc104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/4191832411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc100360 0x7fd8cc100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/4191832411 >> 172.21.15.121:0/4191832411 conn(0x7fd8cc077250 msgr2=0x7fd8cc077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/4191832411 shutdown_connections 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.824+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/4191832411 wait complete. 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 Processor -- start 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 -- start start 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc105430 0x7fd8cc197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc19b600 0x7fd8cc19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:59.828 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc19bff0 con 0x7fd8cc100360 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc19c160 con 0x7fd8cc105430 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d2f4b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc19c2d0 con 0x7fd8cc19b600 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43706/0 (socket says 172.21.15.121:43706) 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d14e8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc19b600 0x7fd8cc19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 -- 172.21.15.121:0/3079435891 learned_addr learned my addr 172.21.15.121:0/3079435891 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 -- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc19b600 msgr2=0x7fd8cc19bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc19b600 0x7fd8cc19bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 -- 172.21.15.121:0/3079435891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc105430 msgr2=0x7fd8cc197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc105430 0x7fd8cc197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.825+0000 7fd8d0ce7640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0009560 con 0x7fd8cc100360 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8d0ce7640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c920 tx=0x7fd8bc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc007c70 con 0x7fd8cc100360 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c4009840 con 0x7fd8cc100360 2024-02-10T06:31:59.829 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd8bc007dd0 con 0x7fd8cc100360 2024-02-10T06:31:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc018660 con 0x7fd8cc100360 2024-02-10T06:31:59.830 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.826+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc1a3bd0 con 0x7fd8cc100360 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.827+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc068110 con 0x7fd8cc100360 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.827+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fd8bc00f450 con 0x7fd8cc100360 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.828+0000 7fd8c9ffb640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 0x7fd8b406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.828+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(821..821 src has 253..821) v4 ==== 9075+0+0 (secure 0 0 0) 0x7fd8bc08e0b0 con 0x7fd8cc100360 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.828+0000 7fd8cbfff640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 0x7fd8b406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:31:59.831 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.828+0000 7fd8cbfff640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 0x7fd8b406fef0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc104eb0 tx=0x7fd8c0009790 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:31:59.833 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:31:59.832+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8bc056e50 con 0x7fd8cc100360 2024-02-10T06:32:00.007 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:00.006+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fd8cc1086c0 con 0x7fd8cc100360 2024-02-10T06:32:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[41707]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[41707]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:00.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[41707]: osdmap e821: 8 total, 8 up, 8 in 2024-02-10T06:32:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[30020]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[30020]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:00.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:31:59 smithi121 ceph-mon[30020]: osdmap e821: 8 total, 8 up, 8 in 2024-02-10T06:32:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:59 smithi146 ceph-mon[33795]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:59 smithi146 ceph-mon[33795]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:00.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:31:59 smithi146 ceph-mon[33795]: osdmap e821: 8 total, 8 up, 8 in 2024-02-10T06:32:00.798 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:00.797+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v822) v1 ==== 221+0+0 (secure 0 0 0) 0x7fd8bc05a510 con 0x7fd8cc100360 2024-02-10T06:32:00.908 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:00.907+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7fd8cc100d10 con 0x7fd8cc100360 2024-02-10T06:32:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:00 smithi121 ceph-mon[41707]: pgmap v1761: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:01.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:00 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:00 smithi121 ceph-mon[30020]: pgmap v1761: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:01.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:00 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:00 smithi146 ceph-mon[33795]: pgmap v1761: 176 pgs: 176 active+clean; 697 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:00 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:01.804 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.803+0000 7fd8c9ffb640 1 -- 172.21.15.121:0/3079435891 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v823) v1 ==== 221+0+0 (secure 0 0 0) 0x7fd8bc05a6f0 con 0x7fd8cc100360 2024-02-10T06:32:01.804 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_bytes = 100 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:32:01.808 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 msgr2=0x7fd8b406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:01.808 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 0x7fd8b406fef0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc104eb0 tx=0x7fd8c0009790 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 msgr2=0x7fd8cc196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00c920 tx=0x7fd8bc00cde0 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 shutdown_connections 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd8b406da30 0x7fd8b406fef0 unknown :-1 s=CLOSED pgs=3557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8cc19b600 0x7fd8cc19bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8cc105430 0x7fd8cc197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 --2- 172.21.15.121:0/3079435891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc100360 0x7fd8cc196d40 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 >> 172.21.15.121:0/3079435891 conn(0x7fd8cc077250 msgr2=0x7fd8cc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:01.809 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 shutdown_connections 2024-02-10T06:32:01.810 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:01.807+0000 7fd8d2f4b640 1 -- 172.21.15.121:0/3079435891 wait complete. 2024-02-10T06:32:01.820 INFO:tasks.workunit.client.0.smithi121.stderr:+ sleep 30 2024-02-10T06:32:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:02.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[41707]: osdmap e822: 8 total, 8 up, 8 in 2024-02-10T06:32:02.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:02.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:02.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[30020]: osdmap e822: 8 total, 8 up, 8 in 2024-02-10T06:32:02.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:01 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:01 smithi146 ceph-mon[33795]: osdmap e822: 8 total, 8 up, 8 in 2024-02-10T06:32:02.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:01 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]: dispatch 2024-02-10T06:32:03.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[30020]: pgmap v1764: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[30020]: osdmap e823: 8 total, 8 up, 8 in 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[41707]: pgmap v1764: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[41707]: osdmap e823: 8 total, 8 up, 8 in 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:03.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:02 smithi146 ceph-mon[33795]: pgmap v1764: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 4.9 KiB/s wr, 1 op/s 2024-02-10T06:32:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:02 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3079435891' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "100"}]': finished 2024-02-10T06:32:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:02 smithi146 ceph-mon[33795]: osdmap e823: 8 total, 8 up, 8 in 2024-02-10T06:32:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:02 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:05.078 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[41707]: pgmap v1766: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 875 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[41707]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_bytes: 100 B) 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[41707]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[41707]: osdmap e824: 8 total, 8 up, 8 in 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[30020]: pgmap v1766: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 875 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[30020]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_bytes: 100 B) 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[30020]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:32:05.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:04 smithi121 ceph-mon[30020]: osdmap e824: 8 total, 8 up, 8 in 2024-02-10T06:32:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:04 smithi146 ceph-mon[33795]: pgmap v1766: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 875 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-02-10T06:32:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:04 smithi146 ceph-mon[33795]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' is full (reached quota's max_bytes: 100 B) 2024-02-10T06:32:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:04 smithi146 ceph-mon[33795]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:32:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:04 smithi146 ceph-mon[33795]: osdmap e824: 8 total, 8 up, 8 in 2024-02-10T06:32:07.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:06 smithi121 ceph-mon[41707]: pgmap v1768: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-10T06:32:07.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:06 smithi121 ceph-mon[30020]: pgmap v1768: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-10T06:32:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:06 smithi146 ceph-mon[33795]: pgmap v1768: 176 pgs: 176 active+clean; 720 KiB data, 859 MiB used, 714 GiB / 715 GiB avail; 861 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-10T06:32:08.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:08 smithi121 ceph-mon[41707]: pgmap v1769: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-02-10T06:32:09.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:08 smithi121 ceph-mon[30020]: pgmap v1769: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-02-10T06:32:09.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:08 smithi146 ceph-mon[33795]: pgmap v1769: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 1 op/s 2024-02-10T06:32:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:11.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:10 smithi121 ceph-mon[41707]: pgmap v1770: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:11.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:10 smithi121 ceph-mon[30020]: pgmap v1770: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:10 smithi146 ceph-mon[33795]: pgmap v1770: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T06:32:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:12 smithi146 ceph-mon[33795]: pgmap v1771: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:12 smithi121 ceph-mon[41707]: pgmap v1771: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:13.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:12 smithi121 ceph-mon[30020]: pgmap v1771: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:14 smithi146 ceph-mon[33795]: pgmap v1772: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:14 smithi121 ceph-mon[41707]: pgmap v1772: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:14 smithi121 ceph-mon[30020]: pgmap v1772: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:16 smithi146 ceph-mon[33795]: pgmap v1773: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:32:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:16 smithi121 ceph-mon[30020]: pgmap v1773: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:32:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:16 smithi121 ceph-mon[41707]: pgmap v1773: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-02-10T06:32:18.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[30020]: pgmap v1774: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:19.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[41707]: pgmap v1774: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:19.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:18 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:18 smithi146 ceph-mon[33795]: pgmap v1774: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:18 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:18 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:21.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:20 smithi121 ceph-mon[41707]: pgmap v1775: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:21.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:20 smithi121 ceph-mon[30020]: pgmap v1775: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:20 smithi146 ceph-mon[33795]: pgmap v1775: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:23.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:22 smithi121 ceph-mon[41707]: pgmap v1776: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:23.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:22 smithi121 ceph-mon[30020]: pgmap v1776: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:22 smithi146 ceph-mon[33795]: pgmap v1776: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:25.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:24 smithi121 ceph-mon[41707]: pgmap v1777: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:25.079 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:25.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:24 smithi121 ceph-mon[30020]: pgmap v1777: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:24 smithi146 ceph-mon[33795]: pgmap v1777: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:27.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:26 smithi121 ceph-mon[41707]: pgmap v1778: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:27.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:26 smithi121 ceph-mon[30020]: pgmap v1778: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:26 smithi146 ceph-mon[33795]: pgmap v1778: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:28.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:29.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:28 smithi121 ceph-mon[41707]: pgmap v1779: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:29.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:28 smithi121 ceph-mon[30020]: pgmap v1779: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:29.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:28 smithi146 ceph-mon[33795]: pgmap v1779: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:31.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:30 smithi121 ceph-mon[41707]: pgmap v1780: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:31.079 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:30 smithi121 ceph-mon[30020]: pgmap v1780: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:30 smithi146 ceph-mon[33795]: pgmap v1780: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:31.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ pid=158722 2024-02-10T06:32:31.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_bytes 0 2024-02-10T06:32:31.822 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put two /etc/passwd 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 -- 172.21.15.121:0/3606550920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054100d30 msgr2=0x7fd054104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 --2- 172.21.15.121:0/3606550920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054100d30 0x7fd054104d70 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fd048009930 tx=0x7fd04802bc30 comp rx=0 tx=0).stop 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 -- 172.21.15.121:0/3606550920 shutdown_connections 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 --2- 172.21.15.121:0/3606550920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054105450 0x7fd05410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 --2- 172.21.15.121:0/3606550920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054100d30 0x7fd054104d70 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 --2- 172.21.15.121:0/3606550920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd054100380 0x7fd054100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 -- 172.21.15.121:0/3606550920 >> 172.21.15.121:0/3606550920 conn(0x7fd054077310 msgr2=0x7fd054077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 -- 172.21.15.121:0/3606550920 shutdown_connections 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 -- 172.21.15.121:0/3606550920 wait complete. 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.914+0000 7fd05940e640 1 Processor -- start 2024-02-10T06:32:31.919 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 -- start start 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054100380 0x7fd054197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 0x7fd0541a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0541a3410 con 0x7fd054105450 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0541a3580 con 0x7fd054100380 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd05940e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0541a3740 con 0x7fd0541987c0 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49958/0 (socket says 172.21.15.121:49958) 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 -- 172.21.15.121:0/732117966 learned_addr learned my addr 172.21.15.121:0/732117966 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0537fe640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 0x7fd0541a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd052ffd640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054100380 0x7fd054197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 -- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 msgr2=0x7fd0541a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:31.920 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 0x7fd0541a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 -- 172.21.15.121:0/732117966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054100380 msgr2=0x7fd054197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054100380 0x7fd054197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044009490 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0537fe640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 0x7fd0541a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.915+0000 7fd0527fc640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd048009230 tx=0x7fd048004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.916+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480035f0 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.916+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd048003bd0 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.916+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd048009560 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.916+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048005550 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.916+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0541a3d40 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.917+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd054068560 con 0x7fd054105450 2024-02-10T06:32:31.921 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.917+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fd048003750 con 0x7fd054105450 2024-02-10T06:32:31.922 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.918+0000 7fd03bfff640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 0x7fd03c06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:31.922 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.918+0000 7fd052ffd640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 0x7fd03c06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:31.922 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.918+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(824..824 src has 253..824) v4 ==== 9075+0+0 (secure 0 0 0) 0x7fd0480b0540 con 0x7fd054105450 2024-02-10T06:32:31.922 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.918+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=825}) v3 -- 0x7fd03c0794c0 con 0x7fd054105450 2024-02-10T06:32:31.922 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.918+0000 7fd052ffd640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 0x7fd03c06fef0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fd044009c10 tx=0x7fd044005f90 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:31.924 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:31.923+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd0480792e0 con 0x7fd054105450 2024-02-10T06:32:32.097 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:32.095+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fd054100d30 con 0x7fd054105450 2024-02-10T06:32:32.825 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:32.824+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v825) v1 ==== 217+0+0 (secure 0 0 0) 0x7fd0480430d0 con 0x7fd054105450 2024-02-10T06:32:32.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:32.825+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 8 ==== osd_map(825..825 src has 253..825) v4 ==== 628+0+0 (secure 0 0 0) 0x7fd048003a00 con 0x7fd054105450 2024-02-10T06:32:32.826 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:32.825+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=826}) v3 -- 0x7fd03c07a3a0 con 0x7fd054105450 2024-02-10T06:32:32.937 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:32.936+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fd054100f40 con 0x7fd054105450 2024-02-10T06:32:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:32 smithi146 ceph-mon[33795]: pgmap v1781: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:32 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[41707]: pgmap v1781: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[30020]: pgmap v1781: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:33.844 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.843+0000 7fd03bfff640 1 -- 172.21.15.121:0/732117966 <== mon.0 v2:172.21.15.121:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v826) v1 ==== 217+0+0 (secure 0 0 0) 0x7fd0480056b0 con 0x7fd054105450 2024-02-10T06:32:33.844 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:32:33.846 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 msgr2=0x7fd03c06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:33.846 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 0x7fd03c06fef0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fd044009c10 tx=0x7fd044005f90 comp rx=0 tx=0).stop 2024-02-10T06:32:33.846 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 msgr2=0x7fd054198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:33.846 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd048009230 tx=0x7fd048004270 comp rx=0 tx=0).stop 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 shutdown_connections 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd03c06da30 0x7fd03c06fef0 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0541987c0 0x7fd0541a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd054105450 0x7fd054198230 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 --2- 172.21.15.121:0/732117966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd054100380 0x7fd054197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.845+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 >> 172.21.15.121:0/732117966 conn(0x7fd054077310 msgr2=0x7fd0540fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.846+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 shutdown_connections 2024-02-10T06:32:33.847 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.846+0000 7fd05940e640 1 -- 172.21.15.121:0/732117966 wait complete. 2024-02-10T06:32:33.860 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_objects 0 2024-02-10T06:32:33.947 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- 172.21.15.121:0/3495157952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101e10 msgr2=0x7f3328105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:33.947 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 --2- 172.21.15.121:0/3495157952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101e10 0x7f3328105e50 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f3314009900 tx=0x7f331402bc30 comp rx=0 tx=0).stop 2024-02-10T06:32:33.947 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- 172.21.15.121:0/3495157952 shutdown_connections 2024-02-10T06:32:33.947 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 --2- 172.21.15.121:0/3495157952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 0x7f332810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.947 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 --2- 172.21.15.121:0/3495157952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101e10 0x7f3328105e50 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 --2- 172.21.15.121:0/3495157952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3328101460 0x7f3328101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- 172.21.15.121:0/3495157952 >> 172.21.15.121:0/3495157952 conn(0x7f33280fccc0 msgr2=0x7f33280ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- 172.21.15.121:0/3495157952 shutdown_connections 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- 172.21.15.121:0/3495157952 wait complete. 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 Processor -- start 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.944+0000 7f332f00e640 1 -- start start 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 0x7f33281973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 0x7f332819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332819c120 con 0x7f332819b730 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332819c290 con 0x7f3328106530 2024-02-10T06:32:33.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332819c400 con 0x7f3328101460 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332d5ab640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 0x7f332819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59522/0 (socket says 172.21.15.121:59522) 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 -- 172.21.15.121:0/2002059996 learned_addr learned my addr 172.21.15.121:0/2002059996 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f3327fff640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 0x7f33281973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 -- 172.21.15.121:0/2002059996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 msgr2=0x7f33281973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 0x7f33281973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 -- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 msgr2=0x7f332819bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 0x7f332819bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331c009490 con 0x7f3328101460 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332d5ab640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 0x7f332819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332cdaa640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f331c00ea60 tx=0x7f331c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:33.949 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c00cde0 con 0x7f3328101460 2024-02-10T06:32:33.950 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f331c0044a0 con 0x7f3328101460 2024-02-10T06:32:33.950 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3314009560 con 0x7f3328101460 2024-02-10T06:32:33.950 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.945+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c010430 con 0x7f3328101460 2024-02-10T06:32:33.950 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.946+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33281a3d00 con 0x7f3328101460 2024-02-10T06:32:33.950 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.947+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3328068560 con 0x7f3328101460 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.947+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f331c003f60 con 0x7f3328101460 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.947+0000 7f3325ffb640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 0x7f331806fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.947+0000 7f3327fff640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 0x7f331806fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.947+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(826..826 src has 253..826) v4 ==== 9075+0+0 (secure 0 0 0) 0x7f331c014070 con 0x7f3328101460 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.948+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=827}) v3 -- 0x7f33180794c0 con 0x7f3328101460 2024-02-10T06:32:33.951 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.948+0000 7f3327fff640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 0x7f331806fef0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f3314036040 tx=0x7f3314005fd0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:33.953 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:33.952+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f331c0571e0 con 0x7f3328101460 2024-02-10T06:32:34.128 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:34.127+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f3328101e10 con 0x7f3328101460 2024-02-10T06:32:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:33 smithi146 ceph-mon[33795]: osdmap e825: 8 total, 8 up, 8 in 2024-02-10T06:32:34.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:33 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[41707]: osdmap e825: 8 total, 8 up, 8 in 2024-02-10T06:32:34.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[30020]: osdmap e825: 8 total, 8 up, 8 in 2024-02-10T06:32:34.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:33 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:32:34.701 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:34.700+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 7 ==== osd_map(827..827 src has 253..827) v4 ==== 628+0+0 (secure 0 0 0) 0x7f331c05b640 con 0x7f3328101460 2024-02-10T06:32:34.701 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:34.700+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=828}) v3 -- 0x7f3318079ff0 con 0x7f3328101460 2024-02-10T06:32:34.705 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:34.705+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v827) v1 ==== 221+0+0 (secure 0 0 0) 0x7f331c05a8a0 con 0x7f3328101460 2024-02-10T06:32:34.813 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:34.813+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f33281981c0 con 0x7f3328101460 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: pgmap v1783: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: osdmap e826: 8 total, 8 up, 8 in 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:35.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:35.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: osdmap e827: 8 total, 8 up, 8 in 2024-02-10T06:32:35.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.173 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:34 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: pgmap v1783: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: osdmap e826: 8 total, 8 up, 8 in 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: osdmap e827: 8 total, 8 up, 8 in 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: pgmap v1783: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/732117966' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: osdmap e826: 8 total, 8 up, 8 in 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' no longer out of quota; removing NO_QUOTA flag 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: osdmap e827: 8 total, 8 up, 8 in 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2002059996' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:34 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:32:35.714 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.713+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 9 ==== osd_map(828..828 src has 253..828) v4 ==== 628+0+0 (secure 0 0 0) 0x7f331c05a8a0 con 0x7f3328101460 2024-02-10T06:32:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.717+0000 7f3325ffb640 1 -- 172.21.15.121:0/2002059996 <== mon.2 v2:172.21.15.121:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v828) v1 ==== 221+0+0 (secure 0 0 0) 0x7f331c05aa80 con 0x7f3328101460 2024-02-10T06:32:35.718 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:32:35.722 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 msgr2=0x7f331806fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:35.722 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 0x7f331806fef0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f3314036040 tx=0x7f3314005fd0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.722 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 msgr2=0x7f3328196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f331c00ea60 tx=0x7f331c00ef20 comp rx=0 tx=0).stop 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 shutdown_connections 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f331806da30 0x7f331806fef0 unknown :-1 s=CLOSED pgs=3560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332819b730 0x7f332819bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3328106530 0x7f33281973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 --2- 172.21.15.121:0/2002059996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3328101460 0x7f3328196e70 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 >> 172.21.15.121:0/2002059996 conn(0x7f33280fccc0 msgr2=0x7f33280fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 shutdown_connections 2024-02-10T06:32:35.723 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.721+0000 7f332f00e640 1 -- 172.21.15.121:0/2002059996 wait complete. 2024-02-10T06:32:35.732 INFO:tasks.workunit.client.0.smithi121.stderr:+ wait 158722 2024-02-10T06:32:35.733 INFO:tasks.workunit.client.0.smithi121.stderr:+ '[' 0 -ne 0 ']' 2024-02-10T06:32:35.733 INFO:tasks.workunit.client.0.smithi121.stderr:+ true 2024-02-10T06:32:35.733 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put three /etc/passwd 2024-02-10T06:32:35.773 INFO:tasks.workunit.client.0.smithi121.stderr:++ uuidgen 2024-02-10T06:32:35.774 INFO:tasks.workunit.client.0.smithi121.stderr:+ pp=f00f7468-5de5-4711-bc30-b9e4a19a4526 2024-02-10T06:32:35.774 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool create f00f7468-5de5-4711-bc30-b9e4a19a4526 12 2024-02-10T06:32:35.861 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 -- 172.21.15.121:0/1419512746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100020 msgr2=0x7fbe18100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/1419512746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100020 0x7fbe18100400 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009900 tx=0x7fbe0802bc30 comp rx=0 tx=0).stop 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 -- 172.21.15.121:0/1419512746 shutdown_connections 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/1419512746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe1810dbd0 0x7fbe1810ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/1419512746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe18100940 0x7fbe1810d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/1419512746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100020 0x7fbe18100400 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 -- 172.21.15.121:0/1419512746 >> 172.21.15.121:0/1419512746 conn(0x7fbe18077250 msgr2=0x7fbe18077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 -- 172.21.15.121:0/1419512746 shutdown_connections 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.858+0000 7fbe1e62e640 1 -- 172.21.15.121:0/1419512746 wait complete. 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 Processor -- start 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 -- start start 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe18100020 0x7fbe18199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:35.862 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 0x7fbe1819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1819dfc0 con 0x7fbe18100940 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1819e130 con 0x7fbe18100020 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1e62e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1819e2f0 con 0x7fbe1810dbd0 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe177fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe177fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50014/0 (socket says 172.21.15.121:50014) 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe177fe640 1 -- 172.21.15.121:0/4248618539 learned_addr learned my addr 172.21.15.121:0/4248618539 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.859+0000 7fbe1cbcb640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 0x7fbe1819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe17fff640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe18100020 0x7fbe18199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 -- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 msgr2=0x7fbe1819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 0x7fbe1819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 -- 172.21.15.121:0/4248618539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe18100020 msgr2=0x7fbe18199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:35.863 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe18100020 0x7fbe18199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c009490 con 0x7fbe18100940 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe1cbcb640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 0x7fbe1819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe177fe640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00ea60 tx=0x7fbe0c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00cde0 con 0x7fbe18100940 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbe0c0044a0 con 0x7fbe18100940 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe08009560 con 0x7fbe18100940 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c010430 con 0x7fbe18100940 2024-02-10T06:32:35.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.860+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe181a5c70 con 0x7fbe18100940 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.861+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe18068110 con 0x7fbe18100940 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.862+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fbe0c003f60 con 0x7fbe18100940 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.862+0000 7fbe157fa640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 0x7fbe0006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.862+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(828..828 src has 253..828) v4 ==== 9075+0+0 (secure 0 0 0) 0x7fbe0c014070 con 0x7fbe18100940 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.862+0000 7fbe17fff640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 0x7fbe0006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:35.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.862+0000 7fbe17fff640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 0x7fbe0006fef0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7fbe08036040 tx=0x7fbe08005fd0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:35.867 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:35.867+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe0c0571d0 con 0x7fbe18100940 2024-02-10T06:32:36.045 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.044+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12} v 0) v1 -- 0x7fbe18112a50 con 0x7fbe18100940 2024-02-10T06:32:36.848 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.847+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]=0 pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' created v829) v1 ==== 176+0+0 (secure 0 0 0) 0x7fbe0c05a890 con 0x7fbe18100940 2024-02-10T06:32:36.960 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.959+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12} v 0) v1 -- 0x7fbe18112be0 con 0x7fbe18100940 2024-02-10T06:32:36.960 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.959+0000 7fbe157fa640 1 -- 172.21.15.121:0/4248618539 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]=0 pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' already exists v829) v1 ==== 183+0+0 (secure 0 0 0) 0x7fbe0c05aa70 con 0x7fbe18100940 2024-02-10T06:32:36.960 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' already exists 2024-02-10T06:32:36.962 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 msgr2=0x7fbe0006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 0x7fbe0006fef0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7fbe08036040 tx=0x7fbe08005fd0 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 msgr2=0x7fbe18199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00ea60 tx=0x7fbe0c00ef20 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 shutdown_connections 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fbe0006da30 0x7fbe0006fef0 unknown :-1 s=CLOSED pgs=3562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe1810dbd0 0x7fbe1819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe18100940 0x7fbe18199640 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 --2- 172.21.15.121:0/4248618539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe18100020 0x7fbe18199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 >> 172.21.15.121:0/4248618539 conn(0x7fbe18077250 msgr2=0x7fbe180ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 shutdown_connections 2024-02-10T06:32:36.963 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:36.961+0000 7fbe1e62e640 1 -- 172.21.15.121:0/4248618539 wait complete. 2024-02-10T06:32:36.971 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool application enable f00f7468-5de5-4711-bc30-b9e4a19a4526 rados 2024-02-10T06:32:37.056 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 -- 172.21.15.121:0/2166260042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100d10 msgr2=0x7fb594104d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 --2- 172.21.15.121:0/2166260042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100d10 0x7fb594104d50 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7fb584009900 tx=0x7fb58402bc30 comp rx=0 tx=0).stop 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 -- 172.21.15.121:0/2166260042 shutdown_connections 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 --2- 172.21.15.121:0/2166260042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb594105430 0x7fb59410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 --2- 172.21.15.121:0/2166260042 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100d10 0x7fb594104d50 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 --2- 172.21.15.121:0/2166260042 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594100360 0x7fb594100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 -- 172.21.15.121:0/2166260042 >> 172.21.15.121:0/2166260042 conn(0x7fb594077250 msgr2=0x7fb594077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 -- 172.21.15.121:0/2166260042 shutdown_connections 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.053+0000 7fb59be64640 1 -- 172.21.15.121:0/2166260042 wait complete. 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 Processor -- start 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 -- start start 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:37.057 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59419b6f0 0x7fb59419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59419c0e0 con 0x7fb594105430 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59419c250 con 0x7fb59419b6f0 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb59be64640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59419c3c0 con 0x7fb594100360 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb599c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50030/0 (socket says 172.21.15.121:50030) 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb599c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:59608/0 (socket says 172.21.15.121:59608) 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 -- 172.21.15.121:0/3660168899 learned_addr learned my addr 172.21.15.121:0/3660168899 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 -- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 msgr2=0x7fb594196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 -- 172.21.15.121:0/3660168899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59419b6f0 msgr2=0x7fb59419bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:37.058 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59419b6f0 0x7fb59419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb590009490 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb599c00640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.054+0000 7fb5993ff640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb584000c00 tx=0x7fb584004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.055+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5840035f0 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.055+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb584009560 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.055+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb584003bd0 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.055+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5840054c0 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.055+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5941a3c90 con 0x7fb594105430 2024-02-10T06:32:37.059 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.056+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb594068110 con 0x7fb594105430 2024-02-10T06:32:37.060 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.056+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fb584003750 con 0x7fb594105430 2024-02-10T06:32:37.060 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.056+0000 7fb58affd640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 0x7fb58006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:37.060 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.057+0000 7fb599c00640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 0x7fb58006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:37.060 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.057+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(829..829 src has 253..829) v4 ==== 9450+0+0 (secure 0 0 0) 0x7fb5840b06b0 con 0x7fb594105430 2024-02-10T06:32:37.060 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.057+0000 7fb599c00640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 0x7fb58006fef0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7fb590011630 tx=0x7fb590005f90 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:37.062 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.061+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb5840792e0 con 0x7fb594105430 2024-02-10T06:32:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[30020]: pgmap v1786: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[30020]: osdmap e828: 8 total, 8 up, 8 in 2024-02-10T06:32:37.078 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:37.078 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[41707]: pgmap v1786: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:37.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:37.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[41707]: osdmap e828: 8 total, 8 up, 8 in 2024-02-10T06:32:37.079 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:36 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:36 smithi146 ceph-mon[33795]: pgmap v1786: 176 pgs: 176 active+clean; 720 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:36 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:32:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:36 smithi146 ceph-mon[33795]: osdmap e828: 8 total, 8 up, 8 in 2024-02-10T06:32:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:36 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:37.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.237+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"} v 0) v1 -- 0x7fb594100d10 con 0x7fb594105430 2024-02-10T06:32:37.853 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.852+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]=0 enabled application 'rados' on pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' v830) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb58407c9a0 con 0x7fb594105430 2024-02-10T06:32:37.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:37.966+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"} v 0) v1 -- 0x7fb594110860 con 0x7fb594105430 2024-02-10T06:32:38.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:38.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]': finished 2024-02-10T06:32:38.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:37 smithi146 ceph-mon[33795]: osdmap e829: 8 total, 8 up, 8 in 2024-02-10T06:32:38.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:38.074 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:37 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]': finished 2024-02-10T06:32:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[41707]: osdmap e829: 8 total, 8 up, 8 in 2024-02-10T06:32:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:38.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]': finished 2024-02-10T06:32:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[30020]: osdmap e829: 8 total, 8 up, 8 in 2024-02-10T06:32:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/4248618539' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pg_num": 12}]: dispatch 2024-02-10T06:32:38.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:37 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:38.859 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.858+0000 7fb58affd640 1 -- 172.21.15.121:0/3660168899 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]=0 enabled application 'rados' on pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' v831) v1 ==== 213+0+0 (secure 0 0 0) 0x7fb584043020 con 0x7fb594105430 2024-02-10T06:32:38.859 INFO:tasks.workunit.client.0.smithi121.stderr:enabled application 'rados' on pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.862+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 msgr2=0x7fb58006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.862+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 0x7fb58006fef0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7fb590011630 tx=0x7fb590005f90 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.862+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 msgr2=0x7fb594197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.862+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb584000c00 tx=0x7fb584004270 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 shutdown_connections 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fb58006da30 0x7fb58006fef0 unknown :-1 s=CLOSED pgs=3563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59419b6f0 0x7fb59419bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb594105430 0x7fb594197370 secure :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fb584000c00 tx=0x7fb584004270 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 --2- 172.21.15.121:0/3660168899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb594100360 0x7fb594196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 >> 172.21.15.121:0/3660168899 conn(0x7fb594077250 msgr2=0x7fb5940fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:38.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 shutdown_connections 2024-02-10T06:32:38.865 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.863+0000 7fb59be64640 1 -- 172.21.15.121:0/3660168899 wait complete. 2024-02-10T06:32:38.875 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota f00f7468-5de5-4711-bc30-b9e4a19a4526 max_objects 10 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- 172.21.15.121:0/2167897626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4105430 msgr2=0x7f04f410d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 --2- 172.21.15.121:0/2167897626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4105430 0x7f04f410d000 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009cd0 tx=0x7f04ec02c0f0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- 172.21.15.121:0/2167897626 shutdown_connections 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 --2- 172.21.15.121:0/2167897626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4105430 0x7f04f410d000 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 --2- 172.21.15.121:0/2167897626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f4100d10 0x7f04f4104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 --2- 172.21.15.121:0/2167897626 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4100360 0x7f04f4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- 172.21.15.121:0/2167897626 >> 172.21.15.121:0/2167897626 conn(0x7f04f4077250 msgr2=0x7f04f4077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- 172.21.15.121:0/2167897626 shutdown_connections 2024-02-10T06:32:38.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- 172.21.15.121:0/2167897626 wait complete. 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 Processor -- start 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.962+0000 7f04fa82e640 1 -- start start 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4105430 0x7f04f4197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 0x7f04f419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f419c080 con 0x7f04f4100360 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f419c1f0 con 0x7f04f419b690 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f419c360 con 0x7f04f4105430 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51022/0 (socket says 172.21.15.121:51022) 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f37fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4105430 0x7f04f4197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 -- 172.21.15.121:0/1960140779 learned_addr learned my addr 172.21.15.121:0/1960140779 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f8dcb640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 0x7f04f419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:38.966 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 -- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4105430 msgr2=0x7f04f4197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4105430 0x7f04f4197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 -- 172.21.15.121:0/1960140779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 msgr2=0x7f04f419bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 0x7f04f419bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e4009490 con 0x7f04f4100360 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f8dcb640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 0x7f04f419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f3fff640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f04e400ea60 tx=0x7f04e400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e400cde0 con 0x7f04f4100360 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f04e40044a0 con 0x7f04f4100360 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec009900 con 0x7f04f4100360 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4010430 con 0x7f04f4100360 2024-02-10T06:32:38.967 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.963+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f41a3c60 con 0x7f04f4100360 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.965+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f4068110 con 0x7f04f4100360 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.965+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f04e4003f60 con 0x7f04f4100360 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.965+0000 7f04f17fa640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 0x7f04dc06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.966+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(831..831 src has 253..831) v4 ==== 9463+0+0 (secure 0 0 0) 0x7f04e4014070 con 0x7f04f4100360 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.966+0000 7f04f37fe640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 0x7f04dc06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:32:38.968 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.966+0000 7f04f37fe640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 0x7f04dc06fef0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f04f4069640 tx=0x7f04e800a2d0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:32:38.971 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:38.970+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f04e40571e0 con 0x7f04f4100360 2024-02-10T06:32:39.147 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:39.146+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f04f4100d10 con 0x7f04f4100360 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: pgmap v1789: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: osdmap e830: 8 total, 8 up, 8 in 2024-02-10T06:32:39.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:38 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: pgmap v1789: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: osdmap e830: 8 total, 8 up, 8 in 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: pgmap v1789: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:39.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: osdmap e830: 8 total, 8 up, 8 in 2024-02-10T06:32:39.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:38 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]: dispatch 2024-02-10T06:32:39.864 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:39.863+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f00f7468-5de5-4711-bc30-b9e4a19a4526 v832) v1 ==== 223+0+0 (secure 0 0 0) 0x7f04e405a8a0 con 0x7f04f4100360 2024-02-10T06:32:39.976 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:39.975+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f04f4100ea0 con 0x7f04f4100360 2024-02-10T06:32:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:39 smithi146 ceph-mon[33795]: osdmap e831: 8 total, 8 up, 8 in 2024-02-10T06:32:40.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:39 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:40.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[41707]: osdmap e831: 8 total, 8 up, 8 in 2024-02-10T06:32:40.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3660168899' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "app": "rados"}]': finished 2024-02-10T06:32:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[30020]: osdmap e831: 8 total, 8 up, 8 in 2024-02-10T06:32:40.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:39 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:40.866 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.865+0000 7f04f17fa640 1 -- 172.21.15.121:0/1960140779 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool f00f7468-5de5-4711-bc30-b9e4a19a4526 v833) v1 ==== 223+0+0 (secure 0 0 0) 0x7f04e405aa80 con 0x7f04f4100360 2024-02-10T06:32:40.867 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_objects = 10 for pool f00f7468-5de5-4711-bc30-b9e4a19a4526 2024-02-10T06:32:40.869 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 msgr2=0x7f04dc06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:40.869 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 0x7f04dc06fef0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f04f4069640 tx=0x7f04e800a2d0 comp rx=0 tx=0).stop 2024-02-10T06:32:40.869 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 msgr2=0x7f04f4196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:32:40.869 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f04e400ea60 tx=0x7f04e400ef20 comp rx=0 tx=0).stop 2024-02-10T06:32:40.869 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 shutdown_connections 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f04dc06da30 0x7f04dc06fef0 unknown :-1 s=CLOSED pgs=3564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04f419b690 0x7f04f419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04f4105430 0x7f04f4197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 --2- 172.21.15.121:0/1960140779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04f4100360 0x7f04f4196dd0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.868+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 >> 172.21.15.121:0/1960140779 conn(0x7f04f4077250 msgr2=0x7f04f4103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.869+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 shutdown_connections 2024-02-10T06:32:40.870 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:32:40.869+0000 7f04fa82e640 1 -- 172.21.15.121:0/1960140779 wait complete. 2024-02-10T06:32:40.883 INFO:tasks.workunit.client.0.smithi121.stderr:+ sleep 30 2024-02-10T06:32:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:40 smithi146 ceph-mon[33795]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:40 smithi146 ceph-mon[33795]: osdmap e832: 8 total, 8 up, 8 in 2024-02-10T06:32:41.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:40 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[41707]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[41707]: osdmap e832: 8 total, 8 up, 8 in 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[30020]: pgmap v1792: 188 pgs: 12 unknown, 176 active+clean; 742 KiB data, 860 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[30020]: osdmap e832: 8 total, 8 up, 8 in 2024-02-10T06:32:41.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:40 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]: dispatch 2024-02-10T06:32:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:42.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:41 smithi146 ceph-mon[33795]: osdmap e833: 8 total, 8 up, 8 in 2024-02-10T06:32:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:41 smithi121 ceph-mon[41707]: osdmap e833: 8 total, 8 up, 8 in 2024-02-10T06:32:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/1960140779' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "field": "max_objects", "val": "10"}]': finished 2024-02-10T06:32:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:41 smithi121 ceph-mon[30020]: osdmap e833: 8 total, 8 up, 8 in 2024-02-10T06:32:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:42 smithi146 ceph-mon[33795]: pgmap v1795: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:43.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:42 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:42 smithi121 ceph-mon[41707]: pgmap v1795: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:42 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:43.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:42 smithi121 ceph-mon[30020]: pgmap v1795: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:43.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:42 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:32:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:43 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:43 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:44.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:43 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:44.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:32:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:32:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:43 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:44 smithi146 ceph-mon[33795]: pgmap v1796: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-02-10T06:32:45.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:44 smithi146 ceph-mon[33795]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:45.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:44 smithi121 ceph-mon[41707]: pgmap v1796: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-02-10T06:32:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:44 smithi121 ceph-mon[41707]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:44 smithi121 ceph-mon[30020]: pgmap v1796: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-02-10T06:32:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:44 smithi121 ceph-mon[30020]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T06:32:45.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:47.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:46 smithi146 ceph-mon[33795]: pgmap v1797: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-10T06:32:47.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:46 smithi121 ceph-mon[30020]: pgmap v1797: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-10T06:32:47.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:46 smithi121 ceph-mon[41707]: pgmap v1797: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-02-10T06:32:48.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:48 smithi146 ceph-mon[33795]: pgmap v1798: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:48 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:49.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:48 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:49.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[41707]: pgmap v1798: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[30020]: pgmap v1798: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:32:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:48 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:32:51.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:50 smithi146 ceph-mon[33795]: pgmap v1799: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:51.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:50 smithi121 ceph-mon[41707]: pgmap v1799: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:51.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:50 smithi121 ceph-mon[30020]: pgmap v1799: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T06:32:53.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:52 smithi146 ceph-mon[33795]: pgmap v1800: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:32:53.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:52 smithi121 ceph-mon[30020]: pgmap v1800: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:32:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:52 smithi121 ceph-mon[41707]: pgmap v1800: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T06:32:55.130 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:54 smithi121 ceph-mon[41707]: pgmap v1801: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:55.130 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:54 smithi121 ceph-mon[30020]: pgmap v1801: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:55.130 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:32:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:32:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:32:55.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:54 smithi146 ceph-mon[33795]: pgmap v1801: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:57.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:56 smithi146 ceph-mon[33795]: pgmap v1802: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:57.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:56 smithi121 ceph-mon[30020]: pgmap v1802: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:57.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:56 smithi121 ceph-mon[41707]: pgmap v1802: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:32:58.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:32:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:32:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:58 smithi146 ceph-mon[33795]: pgmap v1803: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:32:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:58 smithi121 ceph-mon[41707]: pgmap v1803: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:32:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:32:59.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:58 smithi121 ceph-mon[30020]: pgmap v1803: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:32:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:32:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:01.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:00 smithi146 ceph-mon[33795]: pgmap v1804: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:00 smithi121 ceph-mon[41707]: pgmap v1804: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:01.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:00 smithi121 ceph-mon[30020]: pgmap v1804: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:02 smithi146 ceph-mon[33795]: pgmap v1805: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:03.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:02 smithi121 ceph-mon[41707]: pgmap v1805: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:02 smithi121 ceph-mon[30020]: pgmap v1805: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:05.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:04 smithi146 ceph-mon[33795]: pgmap v1806: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:04 smithi121 ceph-mon[41707]: pgmap v1806: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:04 smithi121 ceph-mon[30020]: pgmap v1806: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:05.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:07.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:06 smithi146 ceph-mon[33795]: pgmap v1807: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:07.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:06 smithi121 ceph-mon[30020]: pgmap v1807: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:07.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:06 smithi121 ceph-mon[41707]: pgmap v1807: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:08.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:08 smithi146 ceph-mon[33795]: pgmap v1808: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:09.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:08 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:08 smithi121 ceph-mon[30020]: pgmap v1808: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:09.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:08 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:08 smithi121 ceph-mon[41707]: pgmap v1808: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:08 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:10.885 INFO:tasks.workunit.client.0.smithi121.stderr:++ seq 1 10 2024-02-10T06:33:10.886 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:10.886 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj1 /etc/passwd 2024-02-10T06:33:10.926 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:10.926 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj2 /etc/passwd 2024-02-10T06:33:10.964 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:10.965 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj3 /etc/passwd 2024-02-10T06:33:11.000 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj4 /etc/passwd 2024-02-10T06:33:11.036 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.036 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj5 /etc/passwd 2024-02-10T06:33:11.071 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.071 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj6 /etc/passwd 2024-02-10T06:33:11.105 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.105 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj7 /etc/passwd 2024-02-10T06:33:11.139 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.139 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj8 /etc/passwd 2024-02-10T06:33:11.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:10 smithi146 ceph-mon[33795]: pgmap v1809: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:11.173 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.174 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj9 /etc/passwd 2024-02-10T06:33:11.208 INFO:tasks.workunit.client.0.smithi121.stderr:+ for f in `seq 1 10` 2024-02-10T06:33:11.209 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p f00f7468-5de5-4711-bc30-b9e4a19a4526 put obj10 /etc/passwd 2024-02-10T06:33:11.243 INFO:tasks.workunit.client.0.smithi121.stderr:+ sleep 30 2024-02-10T06:33:11.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:10 smithi121 ceph-mon[41707]: pgmap v1809: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:10 smithi121 ceph-mon[30020]: pgmap v1809: 188 pgs: 188 active+clean; 742 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:13.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:12 smithi146 ceph-mon[33795]: pgmap v1810: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:13.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:12 smithi121 ceph-mon[30020]: pgmap v1810: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:13.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:12 smithi121 ceph-mon[41707]: pgmap v1810: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:15.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:14 smithi146 ceph-mon[33795]: pgmap v1811: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:14 smithi121 ceph-mon[41707]: pgmap v1811: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:14 smithi121 ceph-mon[30020]: pgmap v1811: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:17.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:16 smithi146 ceph-mon[33795]: pgmap v1812: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:17.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:16 smithi121 ceph-mon[41707]: pgmap v1812: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:17.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:16 smithi121 ceph-mon[30020]: pgmap v1812: 188 pgs: 188 active+clean; 754 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 0 op/s 2024-02-10T06:33:18.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:17 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:18.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:18.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:17 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:18.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:17 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:18 smithi146 ceph-mon[33795]: pgmap v1813: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-10T06:33:19.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:18 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:19.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:18 smithi121 ceph-mon[41707]: pgmap v1813: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-10T06:33:19.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:18 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:18 smithi121 ceph-mon[30020]: pgmap v1813: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-10T06:33:19.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:18 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:19 smithi146 ceph-mon[33795]: pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' is full (reached quota's max_objects: 10) 2024-02-10T06:33:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:19 smithi146 ceph-mon[33795]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:33:20.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:19 smithi146 ceph-mon[33795]: osdmap e834: 8 total, 8 up, 8 in 2024-02-10T06:33:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[41707]: pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' is full (reached quota's max_objects: 10) 2024-02-10T06:33:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[41707]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:33:20.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[41707]: osdmap e834: 8 total, 8 up, 8 in 2024-02-10T06:33:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[30020]: pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' is full (reached quota's max_objects: 10) 2024-02-10T06:33:20.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[30020]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-10T06:33:20.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:19 smithi121 ceph-mon[30020]: osdmap e834: 8 total, 8 up, 8 in 2024-02-10T06:33:21.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:20 smithi146 ceph-mon[33795]: pgmap v1814: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-02-10T06:33:21.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:20 smithi121 ceph-mon[41707]: pgmap v1814: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-02-10T06:33:21.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:20 smithi121 ceph-mon[30020]: pgmap v1814: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-02-10T06:33:23.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:22 smithi146 ceph-mon[33795]: pgmap v1816: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:23.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:22 smithi121 ceph-mon[41707]: pgmap v1816: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:23.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:22 smithi121 ceph-mon[30020]: pgmap v1816: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:25.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:24 smithi146 ceph-mon[33795]: pgmap v1817: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:25.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:24 smithi121 ceph-mon[41707]: pgmap v1817: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:25.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:24 smithi121 ceph-mon[30020]: pgmap v1817: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:25.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:25 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:25] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:27.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:26 smithi146 ceph-mon[33795]: pgmap v1818: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:27.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:26 smithi121 ceph-mon[30020]: pgmap v1818: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:27.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:26 smithi121 ceph-mon[41707]: pgmap v1818: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 11 KiB/s wr, 1 op/s 2024-02-10T06:33:28.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:27 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:28 smithi146 ceph-mon[33795]: pgmap v1819: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:29.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:28 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:29.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:28 smithi121 ceph-mon[30020]: pgmap v1819: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:29.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:28 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:29.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:28 smithi121 ceph-mon[41707]: pgmap v1819: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:29.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:28 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:31.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:30 smithi146 ceph-mon[33795]: pgmap v1820: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:31.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:30 smithi121 ceph-mon[41707]: pgmap v1820: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:31.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:30 smithi121 ceph-mon[30020]: pgmap v1820: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T06:33:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:32 smithi146 ceph-mon[33795]: pgmap v1821: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:33:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:32 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:33.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:32 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[41707]: pgmap v1821: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:33:33.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:33.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[30020]: pgmap v1821: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T06:33:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:33.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:32 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:35.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:34 smithi146 ceph-mon[33795]: pgmap v1822: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:35.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:34 smithi121 ceph-mon[41707]: pgmap v1822: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:35.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:34 smithi121 ceph-mon[30020]: pgmap v1822: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:35.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:35 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:35] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:37.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:36 smithi146 ceph-mon[33795]: pgmap v1823: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:37.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:36 smithi121 ceph-mon[41707]: pgmap v1823: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:37.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:36 smithi121 ceph-mon[30020]: pgmap v1823: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:33:38.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:37 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:38 smithi121 ceph-mon[41707]: pgmap v1824: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:39.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:38 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:38 smithi121 ceph-mon[30020]: pgmap v1824: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:39.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:38 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:38 smithi146 ceph-mon[33795]: pgmap v1824: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:39.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:38 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:41.244 INFO:tasks.workunit.client.0.smithi121.stderr:+ rados -p 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 put threemore /etc/passwd 2024-02-10T06:33:41.285 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_bytes 0 2024-02-10T06:33:41.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:40 smithi121 ceph-mon[41707]: pgmap v1825: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:33:41.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:40 smithi121 ceph-mon[30020]: pgmap v1825: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:33:41.376 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 -- 172.21.15.121:0/1692600488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d04105450 msgr2=0x7f8d0410d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:41.376 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/1692600488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d04105450 0x7f8d0410d020 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009ca0 tx=0x7f8cfc02c0f0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 -- 172.21.15.121:0/1692600488 shutdown_connections 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/1692600488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d04105450 0x7f8d0410d020 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/1692600488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04100d30 0x7f8d04104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/1692600488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 -- 172.21.15.121:0/1692600488 >> 172.21.15.121:0/1692600488 conn(0x7f8d04077310 msgr2=0x7f8d04077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 -- 172.21.15.121:0/1692600488 shutdown_connections 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.372+0000 7f8d0c08f640 1 -- 172.21.15.121:0/1692600488 wait complete. 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 Processor -- start 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 -- start start 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 0x7f8d041980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:41.377 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 0x7f8d041a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d041a3300 con 0x7f8d04100380 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d041a3470 con 0x7f8d04105450 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0c08f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d041a3630 con 0x7f8d041986b0 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d09e2b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 0x7f8d041a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d09e2b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39788/0 (socket says 172.21.15.121:39788) 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d09e2b640 1 -- 172.21.15.121:0/3803305060 learned_addr learned my addr 172.21.15.121:0/3803305060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 -- 172.21.15.121:0/3803305060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 msgr2=0x7f8d041980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0962a640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 0x7f8d041980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 0x7f8d041980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 -- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 msgr2=0x7f8d04197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:41.378 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0a62c640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00009490 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d09e2b640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.373+0000 7f8d0962a640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 0x7f8d041980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8d0a62c640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 0x7f8d041a2c10 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc02c700 tx=0x7f8cfc032f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc034c30 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8cfc032980 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cfc009900 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc0045b0 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.374+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d041a3ba0 con 0x7f8d041986b0 2024-02-10T06:33:41.379 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.375+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d04068560 con 0x7f8d041986b0 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.375+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f8cfc03b040 con 0x7f8d041986b0 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.375+0000 7f8cfaffd640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 0x7f8cf006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.376+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(834..834 src has 253..834) v4 ==== 9463+0+0 (secure 0 0 0) 0x7f8cfc0b0590 con 0x7f8d041986b0 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.376+0000 7f8d09e2b640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 0x7f8cf006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.376+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=835}) v3 -- 0x7f8cf0079ce0 con 0x7f8d041986b0 2024-02-10T06:33:41.380 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.376+0000 7f8d09e2b640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 0x7f8cf006fef0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f8d00011690 tx=0x7f8d00005f90 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:41.382 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.380+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cfc079260 con 0x7f8d041986b0 2024-02-10T06:33:41.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:40 smithi146 ceph-mon[33795]: pgmap v1825: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-10T06:33:41.557 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.555+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8d04100d30 con 0x7f8d041986b0 2024-02-10T06:33:41.933 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.931+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 7 ==== osd_map(835..835 src has 253..835) v4 ==== 628+0+0 (secure 0 0 0) 0x7f8cfc034e20 con 0x7f8d041986b0 2024-02-10T06:33:41.933 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.931+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=836}) v3 -- 0x7f8cf007a850 con 0x7f8d041986b0 2024-02-10T06:33:41.936 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:41.934+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v835) v1 ==== 217+0+0 (secure 0 0 0) 0x7f8cfc04b070 con 0x7f8d041986b0 2024-02-10T06:33:42.044 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.042+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8d04100ed0 con 0x7f8d041986b0 2024-02-10T06:33:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:41 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:41 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:41 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:41 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:41 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:41 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:42.934 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.933+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 9 ==== osd_map(836..836 src has 253..836) v4 ==== 628+0+0 (secure 0 0 0) 0x7f8cfc004830 con 0x7f8d041986b0 2024-02-10T06:33:42.935 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.933+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=837}) v3 -- 0x7f8cf007acc0 con 0x7f8d041986b0 2024-02-10T06:33:42.940 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.938+0000 7f8cfaffd640 1 -- 172.21.15.121:0/3803305060 <== mon.2 v2:172.21.15.121:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v836) v1 ==== 217+0+0 (secure 0 0 0) 0x7f8cfc041030 con 0x7f8d041986b0 2024-02-10T06:33:42.940 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_bytes = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:33:42.944 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 msgr2=0x7f8cf006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:42.944 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 0x7f8cf006fef0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f8d00011690 tx=0x7f8d00005f90 comp rx=0 tx=0).stop 2024-02-10T06:33:42.944 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 msgr2=0x7f8d041a2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 0x7f8d041a2c10 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc02c700 tx=0x7f8cfc032f80 comp rx=0 tx=0).stop 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 shutdown_connections 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f8cf006da30 0x7f8cf006fef0 unknown :-1 s=CLOSED pgs=3576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d041986b0 0x7f8d041a2c10 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d04105450 0x7f8d041980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 --2- 172.21.15.121:0/3803305060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d04100380 0x7f8d04197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 >> 172.21.15.121:0/3803305060 conn(0x7f8d04077310 msgr2=0x7f8d040fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 shutdown_connections 2024-02-10T06:33:42.945 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:42.942+0000 7f8d0c08f640 1 -- 172.21.15.121:0/3803305060 wait complete. 2024-02-10T06:33:42.955 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool set-quota 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 max_objects 0 2024-02-10T06:33:43.045 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 -- 172.21.15.121:0/3644014054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 msgr2=0x7f48680ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 --2- 172.21.15.121:0/3644014054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f48680ffd50 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f4858009930 tx=0x7f485802bc30 comp rx=0 tx=0).stop 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 -- 172.21.15.121:0/3644014054 shutdown_connections 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 --2- 172.21.15.121:0/3644014054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 0x7f486810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 --2- 172.21.15.121:0/3644014054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4868100320 0x7f4868105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 --2- 172.21.15.121:0/3644014054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f48680ffd50 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 -- 172.21.15.121:0/3644014054 >> 172.21.15.121:0/3644014054 conn(0x7f4868076640 msgr2=0x7f48680fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 -- 172.21.15.121:0/3644014054 shutdown_connections 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.041+0000 7f486e284640 1 -- 172.21.15.121:0/3644014054 wait complete. 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 Processor -- start 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 -- start start 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:43.046 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 0x7f48681973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486819c110 con 0x7f48680ff970 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486819c280 con 0x7f4868106530 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486e284640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486819c3f0 con 0x7f486819b720 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f4867fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486c821640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f4867fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39810/0 (socket says 172.21.15.121:39810) 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f4867fff640 1 -- 172.21.15.121:0/619839692 learned_addr learned my addr 172.21.15.121:0/619839692 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f486c821640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:60764/0 (socket says 172.21.15.121:60764) 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.042+0000 7f48677fe640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 0x7f48681973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 -- 172.21.15.121:0/619839692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 msgr2=0x7f48681973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 0x7f48681973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:43.047 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 -- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 msgr2=0x7f4868196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485c009490 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f4867fff640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486c821640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f486000cab0 tx=0x7f486000cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486000ae20 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f486000f8d0 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4858009560 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4860018860 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.043+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48681a3ca0 con 0x7f486819b720 2024-02-10T06:33:43.048 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.044+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4868068560 con 0x7f486819b720 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.044+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f486000f450 con 0x7f486819b720 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.045+0000 7f48657fa640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 0x7f485006fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.045+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(836..836 src has 253..836) v4 ==== 9463+0+0 (secure 0 0 0) 0x7f486008e460 con 0x7f486819b720 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.045+0000 7f4867fff640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 0x7f485006fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.045+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=837}) v3 -- 0x7f4850079ce0 con 0x7f486819b720 2024-02-10T06:33:43.049 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.045+0000 7f4867fff640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 0x7f485006fef0 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7f4858000c00 tx=0x7f48580058d0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:33:43.051 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.049+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4860057130 con 0x7f486819b720 2024-02-10T06:33:43.227 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.225+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f4868100320 con 0x7f486819b720 2024-02-10T06:33:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[41707]: pgmap v1826: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:43.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[41707]: osdmap e835: 8 total, 8 up, 8 in 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[30020]: pgmap v1826: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[30020]: osdmap e835: 8 total, 8 up, 8 in 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:42 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:42 smithi146 ceph-mon[33795]: pgmap v1826: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:42 smithi146 ceph-mon[33795]: osdmap e835: 8 total, 8 up, 8 in 2024-02-10T06:33:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:42 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3803305060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:42 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]: dispatch 2024-02-10T06:33:43.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.946+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 7 ==== osd_map(837..837 src has 253..837) v4 ==== 628+0+0 (secure 0 0 0) 0x7f486000fa40 con 0x7f486819b720 2024-02-10T06:33:43.948 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.946+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=838}) v3 -- 0x7f485007a850 con 0x7f486819b720 2024-02-10T06:33:43.952 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:43.951+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v837) v1 ==== 221+0+0 (secure 0 0 0) 0x7f486005a7f0 con 0x7f486819b720 2024-02-10T06:33:44.063 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.062+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f48681004b0 con 0x7f486819b720 2024-02-10T06:33:44.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: osdmap e836: 8 total, 8 up, 8 in 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: osdmap e836: 8 total, 8 up, 8 in 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:43 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_bytes", "val": "0"}]': finished 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: osdmap e836: 8 total, 8 up, 8 in 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:43 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:44.957 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.955+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 9 ==== osd_map(838..838 src has 253..838) v4 ==== 628+0+0 (secure 0 0 0) 0x7f486000f700 con 0x7f486819b720 2024-02-10T06:33:44.957 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.955+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=839}) v3 -- 0x7f485007ae50 con 0x7f486819b720 2024-02-10T06:33:44.962 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.960+0000 7f48657fa640 1 -- 172.21.15.121:0/619839692 <== mon.2 v2:172.21.15.121:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 v838) v1 ==== 221+0+0 (secure 0 0 0) 0x7f4860071020 con 0x7f486819b720 2024-02-10T06:33:44.962 INFO:tasks.workunit.client.0.smithi121.stderr:set-quota max_objects = 0 for pool 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 2024-02-10T06:33:44.964 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.962+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 msgr2=0x7f485006fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:44.964 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 0x7f485006fef0 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7f4858000c00 tx=0x7f48580058d0 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 msgr2=0x7f486819bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f486000cab0 tx=0x7f486000cf70 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 shutdown_connections 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f485006da30 0x7f485006fef0 unknown :-1 s=CLOSED pgs=3577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f486819b720 0x7f486819bbd0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4868106530 0x7f48681973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 --2- 172.21.15.121:0/619839692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48680ff970 0x7f4868196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 >> 172.21.15.121:0/619839692 conn(0x7f4868076640 msgr2=0x7f48681040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 shutdown_connections 2024-02-10T06:33:44.965 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:33:44.963+0000 7f486e284640 1 -- 172.21.15.121:0/619839692 wait complete. 2024-02-10T06:33:44.974 INFO:tasks.workunit.client.0.smithi121.stderr:+ sleep 30 2024-02-10T06:33:45.328 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:45 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:45] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: pgmap v1829: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: osdmap e837: 8 total, 8 up, 8 in 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: pgmap v1829: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:45.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: osdmap e837: 8 total, 8 up, 8 in 2024-02-10T06:33:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:45.330 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:44 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: pgmap v1829: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: osdmap e837: 8 total, 8 up, 8 in 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/619839692' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]: dispatch 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T06:33:45.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:44 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:33:46.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:45 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:46.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:45 smithi121 ceph-mon[30020]: osdmap e838: 8 total, 8 up, 8 in 2024-02-10T06:33:46.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:45 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:46.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:45 smithi121 ceph-mon[41707]: osdmap e838: 8 total, 8 up, 8 in 2024-02-10T06:33:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:45 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "field": "max_objects", "val": "0"}]': finished 2024-02-10T06:33:46.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:45 smithi146 ceph-mon[33795]: osdmap e838: 8 total, 8 up, 8 in 2024-02-10T06:33:47.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:46 smithi121 ceph-mon[41707]: pgmap v1832: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:33:47.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:46 smithi121 ceph-mon[30020]: pgmap v1832: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:33:47.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:46 smithi146 ceph-mon[33795]: pgmap v1832: 188 pgs: 188 active+clean; 857 KiB data, 861 MiB used, 714 GiB / 715 GiB avail 2024-02-10T06:33:48.172 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:47 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:48.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:47 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:48.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:47 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:48.328 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:47 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:33:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:48 smithi121 ceph-mon[41707]: pgmap v1833: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 892 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-02-10T06:33:49.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:48 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:48 smithi121 ceph-mon[30020]: pgmap v1833: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 892 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-02-10T06:33:49.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:48 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:48 smithi146 ceph-mon[33795]: pgmap v1833: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 892 B/s rd, 2.1 KiB/s wr, 1 op/s 2024-02-10T06:33:49.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:48 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:51.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:50 smithi121 ceph-mon[41707]: pgmap v1834: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 760 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-02-10T06:33:51.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:50 smithi121 ceph-mon[30020]: pgmap v1834: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 760 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-02-10T06:33:51.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:50 smithi146 ceph-mon[33795]: pgmap v1834: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 760 B/s rd, 1.8 KiB/s wr, 0 op/s 2024-02-10T06:33:53.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:52 smithi121 ceph-mon[41707]: pgmap v1835: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-02-10T06:33:53.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:52 smithi121 ceph-mon[30020]: pgmap v1835: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-02-10T06:33:53.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:52 smithi146 ceph-mon[33795]: pgmap v1835: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-02-10T06:33:55.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:54 smithi121 ceph-mon[41707]: pgmap v1836: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:33:55.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:33:55 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:33:55] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:33:55.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:54 smithi121 ceph-mon[30020]: pgmap v1836: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:33:55.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:54 smithi146 ceph-mon[33795]: pgmap v1836: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-02-10T06:33:57.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:56 smithi121 ceph-mon[41707]: pgmap v1837: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T06:33:57.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:56 smithi121 ceph-mon[30020]: pgmap v1837: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T06:33:57.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:56 smithi146 ceph-mon[33795]: pgmap v1837: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 955 B/s rd, 1.1 KiB/s wr, 1 op/s 2024-02-10T06:33:58.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:33:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:33:59.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:58 smithi121 ceph-mon[41707]: pgmap v1838: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:59.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:33:58 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:58 smithi121 ceph-mon[30020]: pgmap v1838: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:59.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:33:58 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:33:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:58 smithi146 ceph-mon[33795]: pgmap v1838: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 1 op/s 2024-02-10T06:33:59.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:33:58 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:01.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:00 smithi121 ceph-mon[41707]: pgmap v1839: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:01.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:00 smithi121 ceph-mon[30020]: pgmap v1839: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:01.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:00 smithi146 ceph-mon[33795]: pgmap v1839: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:02 smithi121 ceph-mon[41707]: pgmap v1840: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:03.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:02 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:02 smithi121 ceph-mon[30020]: pgmap v1840: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:03.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:02 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:02 smithi146 ceph-mon[33795]: pgmap v1840: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:03.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:02 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:05.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:04 smithi121 ceph-mon[41707]: pgmap v1841: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:05.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:04 smithi121 ceph-mon[30020]: pgmap v1841: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:05.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:05 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:34:05] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:34:05.421 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:04 smithi146 ceph-mon[33795]: pgmap v1841: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:07.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:06 smithi121 ceph-mon[41707]: pgmap v1842: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:07.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:06 smithi121 ceph-mon[30020]: pgmap v1842: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:07.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:06 smithi146 ceph-mon[33795]: pgmap v1842: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:08.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:34:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:34:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:09 smithi121 ceph-mon[30020]: pgmap v1843: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:09.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:09 smithi121 ceph-mon[30020]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:09 smithi121 ceph-mon[41707]: pgmap v1843: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:09.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:09 smithi121 ceph-mon[41707]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:09 smithi146 ceph-mon[33795]: pgmap v1843: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:09.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:09 smithi146 ceph-mon[33795]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T06:34:11.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:11 smithi121 ceph-mon[30020]: pgmap v1844: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:11.329 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:11 smithi121 ceph-mon[41707]: pgmap v1844: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:11.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:11 smithi146 ceph-mon[33795]: pgmap v1844: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:13.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:13 smithi121 ceph-mon[41707]: pgmap v1845: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:13.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:13 smithi121 ceph-mon[30020]: pgmap v1845: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:13.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:13 smithi146 ceph-mon[33795]: pgmap v1845: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T06:34:14.975 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool delete 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 263f3c8c-b9ac-4c1b-b86f-48ce78064fb6 --yes-i-really-really-mean-it 2024-02-10T06:34:15.074 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 -- 172.21.15.121:0/687485900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8610102d90 msgr2=0x7f861010b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:15.074 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 --2- 172.21.15.121:0/687485900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8610102d90 0x7f861010b270 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f8600009900 tx=0x7f860002bcd0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.074 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 -- 172.21.15.121:0/687485900 shutdown_connections 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 --2- 172.21.15.121:0/687485900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f861010b840 0x7f861010dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 --2- 172.21.15.121:0/687485900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8610102d90 0x7f861010b270 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 --2- 172.21.15.121:0/687485900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 -- 172.21.15.121:0/687485900 >> 172.21.15.121:0/687485900 conn(0x7f86100772b0 msgr2=0x7f86100776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 -- 172.21.15.121:0/687485900 shutdown_connections 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.069+0000 7f86177b6640 1 -- 172.21.15.121:0/687485900 wait complete. 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 Processor -- start 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 -- start start 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 0x7f86101973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:15.075 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 0x7f861019b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861019bd20 con 0x7f8610102470 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861019be90 con 0x7f8610102d90 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f86177b6640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861019c050 con 0x7f861010b840 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f8615552640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f8615552640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43410/0 (socket says 172.21.15.121:43410) 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f8615552640 1 -- 172.21.15.121:0/3666240850 learned_addr learned my addr 172.21.15.121:0/3666240850 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.070+0000 7f8615d53640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 0x7f861019b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8614d51640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 0x7f86101973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:15.076 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 -- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 msgr2=0x7f861019b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 0x7f861019b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 -- 172.21.15.121:0/3666240850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 msgr2=0x7f86101973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 0x7f86101973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c009490 con 0x7f8610102470 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8614d51640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 0x7f86101973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615d53640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 0x7f861019b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f8615552640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f860c00ea60 tx=0x7f860c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c00cde0 con 0x7f8610102470 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f860c0044a0 con 0x7f8610102470 2024-02-10T06:34:15.077 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600009560 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c010430 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.071+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86101a39d0 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.072+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8610068560 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.073+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7f860c003f60 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.073+0000 7f86067fc640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 0x7f85fc06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.073+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(838..838 src has 253..838) v4 ==== 9463+0+0 (secure 0 0 0) 0x7f860c014070 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.073+0000 7f8614d51640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 0x7f85fc06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.073+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=839}) v3 -- 0x7f85fc079ce0 con 0x7f8610102470 2024-02-10T06:34:15.078 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.074+0000 7f8614d51640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 0x7f85fc06fef0 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f8610077c40 tx=0x7f8600005a20 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:15.079 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.078+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f860c0571d0 con 0x7f8610102470 2024-02-10T06:34:15.255 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:15.254+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8610106040 con 0x7f8610102470 2024-02-10T06:34:15.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:15 smithi121 ceph-mon[41707]: pgmap v1846: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:15.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:15 smithi121 ceph-mon[30020]: pgmap v1846: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:15.329 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:15 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: ::ffff:172.21.15.146 - - [10/Feb/2024:06:34:15] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T06:34:15.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:15 smithi146 ceph-mon[33795]: pgmap v1846: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:16.015 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.014+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]=0 pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' removed v839) v1 ==== 248+0+0 (secure 0 0 0) 0x7f860c05a890 con 0x7f8610102470 2024-02-10T06:34:16.017 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.017+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 8 ==== osd_map(839..839 src has 253..839) v4 ==== 296+0+0 (secure 0 0 0) 0x7f860c05b630 con 0x7f8610102470 2024-02-10T06:34:16.017 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.017+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=840}) v3 -- 0x7f85fc07a9c0 con 0x7f8610102470 2024-02-10T06:34:16.141 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.140+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8610110960 con 0x7f8610102470 2024-02-10T06:34:16.141 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.140+0000 7f86067fc640 1 -- 172.21.15.121:0/3666240850 <== mon.0 v2:172.21.15.121:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]=0 pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' does not exist v839) v1 ==== 255+0+0 (secure 0 0 0) 0x7f860c05aa70 con 0x7f8610102470 2024-02-10T06:34:16.141 INFO:tasks.workunit.client.0.smithi121.stderr:pool '263f3c8c-b9ac-4c1b-b86f-48ce78064fb6' does not exist 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 msgr2=0x7f85fc06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 0x7f85fc06fef0 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f8610077c40 tx=0x7f8600005a20 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 msgr2=0x7f8610196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f860c00ea60 tx=0x7f860c00ef20 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 shutdown_connections 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7f85fc06da30 0x7f85fc06fef0 unknown :-1 s=CLOSED pgs=3578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861010b840 0x7f861019b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8610102d90 0x7f86101973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 --2- 172.21.15.121:0/3666240850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8610102470 0x7f8610196e60 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.142+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 >> 172.21.15.121:0/3666240850 conn(0x7f86100772b0 msgr2=0x7f8610103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.143+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 shutdown_connections 2024-02-10T06:34:16.144 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.143+0000 7f86177b6640 1 -- 172.21.15.121:0/3666240850 wait complete. 2024-02-10T06:34:16.152 INFO:tasks.workunit.client.0.smithi121.stderr:+ ceph osd pool delete f00f7468-5de5-4711-bc30-b9e4a19a4526 f00f7468-5de5-4711-bc30-b9e4a19a4526 --yes-i-really-really-mean-it 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.233+0000 7fd2e48c0640 1 -- 172.21.15.121:0/1336365998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0100d20 msgr2=0x7fd2e0104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.233+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/1336365998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0100d20 0x7fd2e0104d60 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009930 tx=0x7fd2c802bc20 comp rx=0 tx=0).stop 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 -- 172.21.15.121:0/1336365998 shutdown_connections 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/1336365998 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 0x7fd2e010d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/1336365998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0100d20 0x7fd2e0104d60 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/1336365998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.237 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 -- 172.21.15.121:0/1336365998 >> 172.21.15.121:0/1336365998 conn(0x7fd2e00772b0 msgr2=0x7fd2e00776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 -- 172.21.15.121:0/1336365998 shutdown_connections 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 -- 172.21.15.121:0/1336365998 wait complete. 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 Processor -- start 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 -- start start 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.234+0000 7fd2e48c0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 0x7fd2e019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 0x7fd2e0194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e01954e0 con 0x7fd2e0194af0 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0195650 con 0x7fd2e0105440 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0195810 con 0x7fd2e0100370 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2ded9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 0x7fd2e0194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:38456/0 (socket says 172.21.15.121:38456) 2024-02-10T06:34:16.238 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 -- 172.21.15.121:0/2004083082 learned_addr learned my addr 172.21.15.121:0/2004083082 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2d7fff640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 0x7fd2e019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 -- 172.21.15.121:0/2004083082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 msgr2=0x7fd2e019ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 0x7fd2e019ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 -- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 msgr2=0x7fd2e0194fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 0x7fd2e0194fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d000b6a0 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2ded9d640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 0x7fd2e0194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2de59c640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0010b80 tx=0x7fd2d000e490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000eb40 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c8009560 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fd2d0011040 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0003ad0 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.235+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e01a1ba0 con 0x7fd2e0100370 2024-02-10T06:34:16.239 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.236+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0068560 con 0x7fd2e0100370 2024-02-10T06:34:16.240 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.237+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 22) v1 ==== 91465+0+0 (secure 0 0 0) 0x7fd2d0003c30 con 0x7fd2e0100370 2024-02-10T06:34:16.240 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.237+0000 7fd2d77fe640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 0x7fd2cc06fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-10T06:34:16.241 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.237+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(839..839 src has 253..839) v4 ==== 9075+0+0 (secure 0 0 0) 0x7fd2d008fcc0 con 0x7fd2e0100370 2024-02-10T06:34:16.241 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.237+0000 7fd2d7fff640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 0x7fd2cc06fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-10T06:34:16.241 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.237+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=840}) v3 -- 0x7fd2cc0794c0 con 0x7fd2e0100370 2024-02-10T06:34:16.241 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.238+0000 7fd2d7fff640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 0x7fd2cc06fef0 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009900 tx=0x7fd2c8005fd0 comp rx=0 tx=0).ready entity=mgr.24455 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T06:34:16.242 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.242+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd2d0058a60 con 0x7fd2e0100370 2024-02-10T06:34:16.328 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:16 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:16.329 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:16 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:16.417 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:16.416+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2e0100d20 con 0x7fd2e0100370 2024-02-10T06:34:16.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:16 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.018 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.017+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 7 ==== osd_map(840..840 src has 253..840) v4 ==== 296+0+0 (secure 0 0 0) 0x7fd2d005cec0 con 0x7fd2e0100370 2024-02-10T06:34:17.018 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.017+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=841}) v3 -- 0x7fd2cc079d10 con 0x7fd2e0100370 2024-02-10T06:34:17.022 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.021+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]=0 pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' removed v840) v1 ==== 248+0+0 (secure 0 0 0) 0x7fd2d001a070 con 0x7fd2e0100370 2024-02-10T06:34:17.135 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.134+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd2e0100f60 con 0x7fd2e0100370 2024-02-10T06:34:17.136 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.135+0000 7fd2d77fe640 1 -- 172.21.15.121:0/2004083082 <== mon.2 v2:172.21.15.121:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]=0 pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' does not exist v840) v1 ==== 255+0+0 (secure 0 0 0) 0x7fd2d005c120 con 0x7fd2e0100370 2024-02-10T06:34:17.136 INFO:tasks.workunit.client.0.smithi121.stderr:pool 'f00f7468-5de5-4711-bc30-b9e4a19a4526' does not exist 2024-02-10T06:34:17.138 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 msgr2=0x7fd2cc06fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:17.138 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 0x7fd2cc06fef0 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009900 tx=0x7fd2c8005fd0 comp rx=0 tx=0).stop 2024-02-10T06:34:17.138 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 msgr2=0x7fd2e019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T06:34:17.138 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0010b80 tx=0x7fd2d000e490 comp rx=0 tx=0).stop 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 shutdown_connections 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:6800/1142815253,v1:172.21.15.121:6801/1142815253] conn(0x7fd2cc06da30 0x7fd2cc06fef0 unknown :-1 s=CLOSED pgs=3579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2e0194af0 0x7fd2e0194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e0105440 0x7fd2e019ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 --2- 172.21.15.121:0/2004083082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2e0100370 0x7fd2e019a940 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 >> 172.21.15.121:0/2004083082 conn(0x7fd2e00772b0 msgr2=0x7fd2e00fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 shutdown_connections 2024-02-10T06:34:17.139 INFO:tasks.workunit.client.0.smithi121.stderr:2024-02-10T06:34:17.137+0000 7fd2e48c0640 1 -- 172.21.15.121:0/2004083082 wait complete. 2024-02-10T06:34:17.147 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo OK 2024-02-10T06:34:17.148 INFO:tasks.workunit.client.0.smithi121.stdout:OK 2024-02-10T06:34:17.148 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-10T06:34:17.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-10T06:34:17.218 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-02-10T06:34:17.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: pgmap v1847: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: osdmap e839: 8 total, 8 up, 8 in 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.282 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: pgmap v1847: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: osdmap e839: 8 total, 8 up, 8 in 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.283 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:17 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: pgmap v1847: 188 pgs: 188 active+clean; 868 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T06:34:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: osdmap e839: 8 total, 8 up, 8 in 2024-02-10T06:34:17.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/3666240850' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "pool2": "263f3c8c-b9ac-4c1b-b86f-48ce78064fb6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.423 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:17.524 DEBUG:teuthology.parallel:result is None 2024-02-10T06:34:17.525 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T06:34:17.592 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T06:34:17.592 DEBUG:teuthology.orchestra.run.smithi121:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-10T06:34:17.651 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-10T06:34:17.651 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T06:34:17.666 INFO:tasks.cephadm:Teardown begin 2024-02-10T06:34:17.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:34:17.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:34:17.753 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T06:34:17.753 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:34:17.772 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T06:34:17.810 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T06:34:17.810 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T06:34:17.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a 2024-02-10T06:34:18.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:34:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug there is no tcmu-runner data available 2024-02-10T06:34:18.123 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: osdmap e840: 8 total, 8 up, 8 in 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[41707]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:17 smithi121 systemd[1]: Stopping Ceph mon.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: osdmap e840: 8 total, 8 up, 8 in 2024-02-10T06:34:18.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:34:18.125 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-mon[30020]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: osdmap e840: 8 total, 8 up, 8 in 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: from='client.? 172.21.15.121:0/2004083082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "pool2": "f00f7468-5de5-4711-bc30-b9e4a19a4526", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: from='mgr.24455 ' entity='mgr.y' 2024-02-10T06:34:18.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:18 smithi146 ceph-mon[33795]: from='mgr.24455 172.21.15.121:0/514716351' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T06:34:18.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a[29997]: 2024-02-10T06:34:18.122+0000 7f8637f73700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:18.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a[29997]: 2024-02-10T06:34:18.122+0000 7f8637f73700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T06:34:18.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 podman[159698]: 2024-02-10 06:34:18.296061508 +0000 UTC m=+0.298368340 container died 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T06:34:19.023 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 podman[159698]: 2024-02-10 06:34:18.820788739 +0000 UTC m=+0.823095570 container cleanup 859646f2eeb7d0b90bece0f424d4b3754bde8a17b50ba36ba2b68aaef4d7feb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True) 2024-02-10T06:34:19.023 INFO:journalctl@ceph.mon.a.smithi121.stdout:Feb 10 06:34:18 smithi121 bash[159698]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T06:34:19.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T06:34:19.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:19.278 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T06:34:19.278 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T06:34:19.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.c 2024-02-10T06:34:19.578 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 systemd[1]: Stopping Ceph mon.c for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:19.893 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c[41684]: 2024-02-10T06:34:19.610+0000 7f3d15c3b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:19.893 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c[41684]: 2024-02-10T06:34:19.610+0000 7f3d15c3b700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T06:34:19.893 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 podman[159830]: 2024-02-10 06:34:19.652113435 +0000 UTC m=+0.159250026 container died 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T06:34:19.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE Bus STOPPING 2024-02-10T06:34:19.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T06:34:19.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE Bus STOPPED 2024-02-10T06:34:19.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE Bus STARTING 2024-02-10T06:34:19.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE Serving on http://:::9283 2024-02-10T06:34:19.895 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:19 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: [10/Feb/2024:06:34:19] ENGINE Bus STARTED 2024-02-10T06:34:20.193 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 podman[159830]: 2024-02-10 06:34:19.983593387 +0000 UTC m=+0.490730026 container cleanup 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:34:20.193 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:19 smithi121 bash[159830]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c 2024-02-10T06:34:20.444 INFO:journalctl@ceph.mon.c.smithi121.stdout:Feb 10 06:34:20 smithi121 podman[159860]: 2024-02-10 06:34:20.193171304 +0000 UTC m=+0.536234929 container remove 429bf4af24cd91e14d8c027cdb777bf2dba8917db14bdbbcd3813f8284a7d2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T06:34:20.447 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T06:34:20.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:20.482 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T06:34:20.482 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T06:34:20.483 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.b 2024-02-10T06:34:20.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:20 smithi146 systemd[1]: Stopping Ceph mon.b for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b[33772]: 2024-02-10T06:34:20.871+0000 7f43591c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b[33772]: 2024-02-10T06:34:20.871+0000 7f43591c4700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T06:34:21.171 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:20 smithi146 podman[79619]: 2024-02-10 06:34:20.892515112 +0000 UTC m=+0.214878598 container died 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:34:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:21 smithi146 podman[79619]: 2024-02-10 06:34:21.361687142 +0000 UTC m=+0.684050675 container cleanup 911367babd6d23d662e3d5fbfd4d0851758a4e0ab1355e5e3e97c807aa36f7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:34:21.422 INFO:journalctl@ceph.mon.b.smithi146.stdout:Feb 10 06:34:21 smithi146 bash[79619]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mon-b 2024-02-10T06:34:21.822 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T06:34:21.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:21.873 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T06:34:21.874 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T06:34:21.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y 2024-02-10T06:34:22.317 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:21 smithi121 systemd[1]: Stopping Ceph mgr.y for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:22.578 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T06:34:22.316+0000 7f35f16fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:22.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:22 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y[30281]: 2024-02-10T06:34:22.316+0000 7f35f16fb700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T06:34:22.579 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:22 smithi121 podman[159973]: 2024-02-10 06:34:22.348427262 +0000 UTC m=+0.280231243 container died d3fdc87b1771ff349ba201da7601f37107503da602f39a2ce48d5f784dd1e0c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:34:22.998 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:22 smithi121 podman[159973]: 2024-02-10 06:34:22.71305589 +0000 UTC m=+0.644859873 container cleanup d3fdc87b1771ff349ba201da7601f37107503da602f39a2ce48d5f784dd1e0c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T06:34:22.998 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Feb 10 06:34:22 smithi121 bash[159973]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-y 2024-02-10T06:34:23.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T06:34:23.166 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:23.167 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T06:34:23.167 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T06:34:23.167 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.x 2024-02-10T06:34:23.509 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 06:34:23 smithi146 systemd[1]: Stopping Ceph mgr.x for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:23.800 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 06:34:23 smithi146 podman[79752]: 2024-02-10 06:34:23.508418974 +0000 UTC m=+0.146601846 container died 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-10T06:34:24.125 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 06:34:23 smithi146 podman[79752]: 2024-02-10 06:34:23.891058675 +0000 UTC m=+0.529241554 container cleanup 32b885adc7ac25a27d53e9481a1106bda6a65caf3ecf11d88a1b3c7b4b1bf847 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:34:24.126 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Feb 10 06:34:23 smithi146 bash[79752]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-mgr-x 2024-02-10T06:34:24.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T06:34:24.293 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:24.293 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T06:34:24.294 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T06:34:24.294 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.0 2024-02-10T06:34:24.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:24 smithi121 systemd[1]: Stopping Ceph osd.0 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:25.078 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:24 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T06:34:24.602+0000 7f117c5ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:25.078 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:24 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T06:34:24.602+0000 7f117c5ab700 -1 osd.0 840 *** Got signal Terminated *** 2024-02-10T06:34:25.078 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:24 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0[46095]: 2024-02-10T06:34:24.602+0000 7f117c5ab700 -1 osd.0 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:34:30.202 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:29 smithi121 podman[160108]: 2024-02-10 06:34:29.943873337 +0000 UTC m=+5.459072982 container died 7e474d56d884e3672882ad8d39f4a7b5ee4a0aad02ab3665ff98d07167fdf6a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T06:34:30.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:30 smithi121 podman[160108]: 2024-02-10 06:34:30.29157079 +0000 UTC m=+5.806770434 container cleanup 7e474d56d884e3672882ad8d39f4a7b5ee4a0aad02ab3665ff98d07167fdf6a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T06:34:30.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:30 smithi121 bash[160108]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0 2024-02-10T06:34:30.907 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:30 smithi121 podman[160143]: 2024-02-10 06:34:30.592966038 +0000 UTC m=+0.651055779 container remove 7e474d56d884e3672882ad8d39f4a7b5ee4a0aad02ab3665ff98d07167fdf6a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T06:34:31.246 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:30 smithi121 podman[160237]: 2024-02-10 06:34:30.810276039 +0000 UTC m=+0.017762705 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:34:31.246 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:30 smithi121 podman[160237]: 2024-02-10 06:34:30.927158622 +0000 UTC m=+0.134645274 container create a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:34:31.246 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:31 smithi121 podman[160237]: 2024-02-10 06:34:31.146129313 +0000 UTC m=+0.353615970 container init a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True) 2024-02-10T06:34:31.247 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:31 smithi121 podman[160237]: 2024-02-10 06:34:31.15168278 +0000 UTC m=+0.359169435 container start a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-10T06:34:31.247 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:31 smithi121 podman[160237]: 2024-02-10 06:34:31.24605337 +0000 UTC m=+0.453540048 container attach a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T06:34:31.578 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:31 smithi121 podman[160237]: 2024-02-10 06:34:31.440503739 +0000 UTC m=+0.647990403 container died a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T06:34:32.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T06:34:32.043 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:31 smithi121 podman[160283]: 2024-02-10 06:34:31.99964982 +0000 UTC m=+0.554195933 container remove a69a64edeee11ab146ee5dddce4455f389dc8f579ff9986ef52bf3ee19ae76fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:34:32.043 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:32 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T06:34:32.043 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:32 smithi121 systemd[1]: Stopped Ceph osd.0 for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:34:32.043 INFO:journalctl@ceph.osd.0.smithi121.stdout:Feb 10 06:34:32 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.0.service: Consumed 54.235s CPU time. 2024-02-10T06:34:32.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:32.060 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T06:34:32.060 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T06:34:32.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.1 2024-02-10T06:34:32.328 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:32 smithi121 systemd[1]: Stopping Ceph osd.1 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:32.829 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:32 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T06:34:32.429+0000 7fe417777700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:32.829 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:32 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T06:34:32.429+0000 7fe417777700 -1 osd.1 840 *** Got signal Terminated *** 2024-02-10T06:34:32.829 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:32 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1[52503]: 2024-02-10T06:34:32.429+0000 7fe417777700 -1 osd.1 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:34:38.260 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:37 smithi121 podman[160369]: 2024-02-10 06:34:37.957055472 +0000 UTC m=+5.675590726 container died 1f47e3e66d9dccad5c16af5710cb4a7d29c2c783135d7aaec527cece4d3d061a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T06:34:38.260 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:38 smithi121 podman[160369]: 2024-02-10 06:34:38.260211297 +0000 UTC m=+5.978746548 container cleanup 1f47e3e66d9dccad5c16af5710cb4a7d29c2c783135d7aaec527cece4d3d061a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T06:34:38.511 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:38 smithi121 bash[160369]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1 2024-02-10T06:34:38.828 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:38 smithi121 podman[160394]: 2024-02-10 06:34:38.511738906 +0000 UTC m=+0.552152604 container remove 1f47e3e66d9dccad5c16af5710cb4a7d29c2c783135d7aaec527cece4d3d061a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:34:39.164 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:38 smithi121 podman[160490]: 2024-02-10 06:34:38.854147767 +0000 UTC m=+0.099482527 container create dcc44e2fc9efbc20af64cac46425e68f1ed13a987ac266873b3ac895dd13ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T06:34:39.165 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:38 smithi121 podman[160490]: 2024-02-10 06:34:38.772100643 +0000 UTC m=+0.017435412 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:34:39.165 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:39 smithi121 podman[160490]: 2024-02-10 06:34:39.081514611 +0000 UTC m=+0.326849381 container init dcc44e2fc9efbc20af64cac46425e68f1ed13a987ac266873b3ac895dd13ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T06:34:39.165 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:39 smithi121 podman[160490]: 2024-02-10 06:34:39.08682553 +0000 UTC m=+0.332160293 container start dcc44e2fc9efbc20af64cac46425e68f1ed13a987ac266873b3ac895dd13ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T06:34:39.165 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:39 smithi121 podman[160490]: 2024-02-10 06:34:39.164548849 +0000 UTC m=+0.409883620 container attach dcc44e2fc9efbc20af64cac46425e68f1ed13a987ac266873b3ac895dd13ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-10T06:34:39.578 INFO:journalctl@ceph.osd.1.smithi121.stdout:Feb 10 06:34:39 smithi121 podman[160490]: 2024-02-10 06:34:39.37147663 +0000 UTC m=+0.616811400 container died dcc44e2fc9efbc20af64cac46425e68f1ed13a987ac266873b3ac895dd13ae9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T06:34:40.041 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T06:34:40.081 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:40.081 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T06:34:40.082 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T06:34:40.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.2 2024-02-10T06:34:40.328 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:40 smithi121 systemd[1]: Stopping Ceph osd.2 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:40.828 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:40 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2[58537]: 2024-02-10T06:34:40.439+0000 7f6a6c631700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:40.828 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:40 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2[58537]: 2024-02-10T06:34:40.439+0000 7f6a6c631700 -1 osd.2 840 *** Got signal Terminated *** 2024-02-10T06:34:40.829 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:40 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2[58537]: 2024-02-10T06:34:40.439+0000 7f6a6c631700 -1 osd.2 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:34:43.734 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:34:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:34:43.436Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.121:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.121:8765: connect: connection refused" 2024-02-10T06:34:43.734 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:34:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:34:43.437Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.121:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.121:8765: connect: connection refused" 2024-02-10T06:34:43.734 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:34:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:34:43.439Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.121:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.121:8765: connect: connection refused" 2024-02-10T06:34:43.735 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:34:43 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:34:43.440Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.121:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.121:8765: connect: connection refused" 2024-02-10T06:34:46.138 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:45 smithi121 podman[160621]: 2024-02-10 06:34:45.886458587 +0000 UTC m=+5.579597866 container died 75904bd5d95d29e936e21359d5232ad7c48cfe305d133c8c98e8c7d915ca660d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:34:46.405 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:46 smithi121 podman[160621]: 2024-02-10 06:34:46.228361226 +0000 UTC m=+5.921500507 container cleanup 75904bd5d95d29e936e21359d5232ad7c48cfe305d133c8c98e8c7d915ca660d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:34:46.405 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:46 smithi121 bash[160621]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2 2024-02-10T06:34:46.828 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:46 smithi121 podman[160645]: 2024-02-10 06:34:46.404724669 +0000 UTC m=+0.514029882 container remove 75904bd5d95d29e936e21359d5232ad7c48cfe305d133c8c98e8c7d915ca660d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T06:34:47.150 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:46 smithi121 podman[160743]: 2024-02-10 06:34:46.738867254 +0000 UTC m=+0.017921703 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:34:47.150 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:46 smithi121 podman[160743]: 2024-02-10 06:34:46.847124089 +0000 UTC m=+0.126178526 container create 101035d789da26adc94e5d4ffe339874ec1b1d5259e3e4ad3f6dbd0b0df83f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T06:34:47.150 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:47 smithi121 podman[160743]: 2024-02-10 06:34:47.057833191 +0000 UTC m=+0.336887635 container init 101035d789da26adc94e5d4ffe339874ec1b1d5259e3e4ad3f6dbd0b0df83f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T06:34:47.150 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:47 smithi121 podman[160743]: 2024-02-10 06:34:47.06311271 +0000 UTC m=+0.342167150 container start 101035d789da26adc94e5d4ffe339874ec1b1d5259e3e4ad3f6dbd0b0df83f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T06:34:47.150 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:47 smithi121 podman[160743]: 2024-02-10 06:34:47.149387979 +0000 UTC m=+0.428442416 container attach 101035d789da26adc94e5d4ffe339874ec1b1d5259e3e4ad3f6dbd0b0df83f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-10T06:34:47.578 INFO:journalctl@ceph.osd.2.smithi121.stdout:Feb 10 06:34:47 smithi121 podman[160743]: 2024-02-10 06:34:47.352495126 +0000 UTC m=+0.631549568 container died 101035d789da26adc94e5d4ffe339874ec1b1d5259e3e4ad3f6dbd0b0df83f30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a) 2024-02-10T06:34:47.963 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T06:34:48.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:48.005 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T06:34:48.005 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T06:34:48.005 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.3 2024-02-10T06:34:48.317 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:48 smithi121 systemd[1]: Stopping Ceph osd.3 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:48.578 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T06:34:48.315+0000 7f79cac60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:48.578 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T06:34:48.315+0000 7f79cac60700 -1 osd.3 840 *** Got signal Terminated *** 2024-02-10T06:34:48.578 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:48 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T06:34:48.315+0000 7f79cac60700 -1 osd.3 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:34:51.672 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:51 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:51.355+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:52.328 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:51 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T06:34:51.850+0000 7f79c7254700 -1 osd.3 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.626026+0000 front 2024-02-10T06:34:28.626189+0000 (oldest deadline 2024-02-10T06:34:51.526173+0000) 2024-02-10T06:34:52.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:52 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:52.347+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:52.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:52 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:52.174+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:52.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:52 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:52.154+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:53.108 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:52 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:52.733+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:53.328 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:52 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3[64710]: 2024-02-10T06:34:52.846+0000 7f79c7254700 -1 osd.3 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.626026+0000 front 2024-02-10T06:34:28.626189+0000 (oldest deadline 2024-02-10T06:34:51.526173+0000) 2024-02-10T06:34:53.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:53 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:53.130+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:53.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:53 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:53.340+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:53.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:53 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:53.107+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:54.077 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:53 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:53.750+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:54.078 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:53 smithi121 podman[160874]: 2024-02-10 06:34:53.766993504 +0000 UTC m=+5.546020051 container died ad93d5ca39ad9a0f5acf05b573b62b16b10967675ef9be671b388b0386d0e694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T06:34:54.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:54 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:54.297+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:54.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:54 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:54.076+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:54.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:54 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:54.138+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:54.578 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160874]: 2024-02-10 06:34:54.079127358 +0000 UTC m=+5.858153906 container cleanup ad93d5ca39ad9a0f5acf05b573b62b16b10967675ef9be671b388b0386d0e694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T06:34:54.578 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 bash[160874]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3 2024-02-10T06:34:54.579 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160898]: 2024-02-10 06:34:54.322657204 +0000 UTC m=+0.555740086 container remove ad93d5ca39ad9a0f5acf05b573b62b16b10967675ef9be671b388b0386d0e694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:34:55.014 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:54 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:54.730+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:55.025 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160997]: 2024-02-10 06:34:54.651707868 +0000 UTC m=+0.017663104 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:34:55.025 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160997]: 2024-02-10 06:34:54.756582037 +0000 UTC m=+0.122537272 container create 85ef8fa3214ae454ab9e973354e88ecdc7578ba2481382b509508b3c3a709dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T06:34:55.025 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160997]: 2024-02-10 06:34:54.942080921 +0000 UTC m=+0.308036162 container init 85ef8fa3214ae454ab9e973354e88ecdc7578ba2481382b509508b3c3a709dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T06:34:55.025 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:54 smithi121 podman[160997]: 2024-02-10 06:34:54.947244822 +0000 UTC m=+0.313200058 container start 85ef8fa3214ae454ab9e973354e88ecdc7578ba2481382b509508b3c3a709dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:34:55.025 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:55 smithi121 podman[160997]: 2024-02-10 06:34:55.025219831 +0000 UTC m=+0.391175080 container attach 85ef8fa3214ae454ab9e973354e88ecdc7578ba2481382b509508b3c3a709dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T06:34:55.320 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:55.113+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:55.320 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:55.093+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:55.328 INFO:journalctl@ceph.osd.3.smithi121.stdout:Feb 10 06:34:55 smithi121 podman[160997]: 2024-02-10 06:34:55.238094525 +0000 UTC m=+0.604049770 container died 85ef8fa3214ae454ab9e973354e88ecdc7578ba2481382b509508b3c3a709dd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-10T06:34:55.672 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:55.318+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:55.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T06:34:55.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:34:55.925 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T06:34:55.925 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T06:34:55.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.4 2024-02-10T06:34:55.957 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:55 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:55.702+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:56.262 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:56.119+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:56.263 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:56.094+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:56.263 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:55 smithi146 systemd[1]: Stopping Ceph osd.4 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:34:56.263 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:56.261+0000 7f7a2b552700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:34:56.263 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:56.261+0000 7f7a2b552700 -1 osd.4 840 *** Got signal Terminated *** 2024-02-10T06:34:56.263 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:56.261+0000 7f7a2b552700 -1 osd.4 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:34:56.672 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:56.279+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:57.130 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:56 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:56.747+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:57.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:57.251+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:57.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:57.136+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:57.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:57.129+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:58.152 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:57 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:57.707+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:58.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:58.245+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:58.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:58.166+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:58.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:58.151+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:59.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:58.734+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:34:59.151 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:58 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:58.734+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.191691+0000 front 2024-02-10T06:34:32.191613+0000 (oldest deadline 2024-02-10T06:34:58.091493+0000) 2024-02-10T06:34:59.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:34:59.209+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:34:59.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:34:59.199+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:34:59.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:59.150+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:34:59.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:34:59.150+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:00.148 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:59.733+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:35:00.148 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:34:59 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:34:59.733+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.191691+0000 front 2024-02-10T06:34:32.191613+0000 (oldest deadline 2024-02-10T06:34:58.091493+0000) 2024-02-10T06:35:00.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:00.198+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:00.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:00.155+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:00.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:00.148+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:00.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:00.148+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:01.172 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:35:00.700+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.091384+0000 front 2024-02-10T06:34:28.091404+0000 (oldest deadline 2024-02-10T06:34:52.191145+0000) 2024-02-10T06:35:01.172 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:00 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4[39220]: 2024-02-10T06:35:00.700+0000 7f7a27b46700 -1 osd.4 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.191691+0000 front 2024-02-10T06:34:32.191613+0000 (oldest deadline 2024-02-10T06:34:58.091493+0000) 2024-02-10T06:35:01.602 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:01.205+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:01.602 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:01.205+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:01.602 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:01.177+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:01.603 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:01.194+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:01.603 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:01 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:01.194+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:01.893 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:01 smithi146 podman[79901]: 2024-02-10 06:35:01.602082607 +0000 UTC m=+5.479188205 container died d1dee2fb77811536d44116fd355019ec2ca281743e72a140c9a799a0aa57d00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-10T06:35:02.172 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:01 smithi146 podman[79901]: 2024-02-10 06:35:01.984027936 +0000 UTC m=+5.861133540 container cleanup d1dee2fb77811536d44116fd355019ec2ca281743e72a140c9a799a0aa57d00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:35:02.172 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:01 smithi146 bash[79901]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4 2024-02-10T06:35:02.552 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[79923]: 2024-02-10 06:35:02.226857224 +0000 UTC m=+0.617967117 container remove d1dee2fb77811536d44116fd355019ec2ca281743e72a140c9a799a0aa57d00a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T06:35:02.552 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:02.190+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:02.552 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:02.190+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:02.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:02.200+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:02.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:02.200+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:02.553 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:02.196+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:02.553 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:02 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:02.196+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:02.838 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[80023]: 2024-02-10 06:35:02.455900489 +0000 UTC m=+0.018697031 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:35:02.839 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[80023]: 2024-02-10 06:35:02.586130086 +0000 UTC m=+0.148926636 container create 6059734d7a0208f72d60bf4c85414260839bf01626287b3787cef98c065f1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T06:35:03.139 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[80023]: 2024-02-10 06:35:02.838632762 +0000 UTC m=+0.401429304 container init 6059734d7a0208f72d60bf4c85414260839bf01626287b3787cef98c065f1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-02-10T06:35:03.139 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[80023]: 2024-02-10 06:35:02.84412125 +0000 UTC m=+0.406917799 container start 6059734d7a0208f72d60bf4c85414260839bf01626287b3787cef98c065f1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T06:35:03.139 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:02 smithi146 podman[80023]: 2024-02-10 06:35:02.930013927 +0000 UTC m=+0.492810474 container attach 6059734d7a0208f72d60bf4c85414260839bf01626287b3787cef98c065f1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:35:03.422 INFO:journalctl@ceph.osd.4.smithi146.stdout:Feb 10 06:35:03 smithi146 podman[80023]: 2024-02-10 06:35:03.139077803 +0000 UTC m=+0.701874338 container died 6059734d7a0208f72d60bf4c85414260839bf01626287b3787cef98c065f1dda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T06:35:03.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:03.196+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:03.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:03.196+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:03.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:03.242+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:03.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:03.242+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:03.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:03.195+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:03.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:03 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:03.195+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:03.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T06:35:03.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:03.959 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T06:35:03.959 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T06:35:03.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.5 2024-02-10T06:35:04.166 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 systemd[1]: Stopping Ceph osd.5 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:04.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:04.189+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:04.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:04.189+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:04.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:04.165+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:04.165+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:04.272+0000 7f482ac5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:04.272+0000 7f482ac5b700 -1 osd.5 840 *** Got signal Terminated *** 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:04.272+0000 7f482ac5b700 -1 osd.5 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:04.205+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:04.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:04 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:04.205+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:05.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:05.119+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:05.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:05.119+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:05.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:05.164+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:05.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:05.164+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:05.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:05.193+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:05.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:05 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:05.193+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:06.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:06.070+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:06.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:06.070+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:06.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:06.210+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:06.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:06.210+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:06.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:06.160+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:06.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:06.160+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:06.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:06 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:06.160+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:07.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:07.242+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:07.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:07.242+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:07.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:07.097+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:07.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:07.097+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:07.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:07.097+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.596103+0000 front 2024-02-10T06:34:41.596203+0000 (oldest deadline 2024-02-10T06:35:06.796510+0000) 2024-02-10T06:35:07.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:07.206+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:07.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:07.206+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:07.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:07 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:07.206+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:08.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:08.277+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:08.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:08.277+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:08.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:08.277+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:08.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:08.134+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:08.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:08.134+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:08.423 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:08.134+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.596103+0000 front 2024-02-10T06:34:41.596203+0000 (oldest deadline 2024-02-10T06:35:06.796510+0000) 2024-02-10T06:35:08.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:08.229+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:08.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:08.229+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:08.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:08 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:08.229+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:09.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:09.142+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.493787+0000 front 2024-02-10T06:34:29.493776+0000 (oldest deadline 2024-02-10T06:34:50.593543+0000) 2024-02-10T06:35:09.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:09.142+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:32.294585+0000 front 2024-02-10T06:34:32.294505+0000 (oldest deadline 2024-02-10T06:35:00.494927+0000) 2024-02-10T06:35:09.422 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5[45195]: 2024-02-10T06:35:09.142+0000 7f4823247700 -1 osd.5 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.596103+0000 front 2024-02-10T06:34:41.596203+0000 (oldest deadline 2024-02-10T06:35:06.796510+0000) 2024-02-10T06:35:09.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:09.260+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:09.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:09.260+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:09.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:09.260+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:09.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:09.304+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:09.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:09.304+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:09.423 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:09 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:09.304+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:09.922 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 podman[80154]: 2024-02-10 06:35:09.638917261 +0000 UTC m=+5.494975130 container died 4fe238241b1f1f809295a3bbb136e1fc20aa30314df0fc84fe2ce4cd3f51ec98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T06:35:10.296 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 podman[80154]: 2024-02-10 06:35:09.969499114 +0000 UTC m=+5.825556985 container cleanup 4fe238241b1f1f809295a3bbb136e1fc20aa30314df0fc84fe2ce4cd3f51ec98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T06:35:10.296 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:09 smithi146 bash[80154]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5 2024-02-10T06:35:10.296 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80176]: 2024-02-10 06:35:10.212155641 +0000 UTC m=+0.574825705 container remove 4fe238241b1f1f809295a3bbb136e1fc20aa30314df0fc84fe2ce4cd3f51ec98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T06:35:10.656 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80274]: 2024-02-10 06:35:10.426093299 +0000 UTC m=+0.017376318 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:35:10.656 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80274]: 2024-02-10 06:35:10.538127481 +0000 UTC m=+0.129410534 container create 8e9f2f1fa8b8caa540080af6f8c39836a1a5a06241137039269a3b8d5365ad92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:10.292+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:10.292+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:10.292+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:10.294+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:10.294+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:10.657 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:10 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:10.294+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:10.915 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80274]: 2024-02-10 06:35:10.824201198 +0000 UTC m=+0.415484209 container init 8e9f2f1fa8b8caa540080af6f8c39836a1a5a06241137039269a3b8d5365ad92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T06:35:10.916 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80274]: 2024-02-10 06:35:10.829964933 +0000 UTC m=+0.421247946 container start 8e9f2f1fa8b8caa540080af6f8c39836a1a5a06241137039269a3b8d5365ad92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T06:35:11.172 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:10 smithi146 podman[80274]: 2024-02-10 06:35:10.915486338 +0000 UTC m=+0.506769364 container attach 8e9f2f1fa8b8caa540080af6f8c39836a1a5a06241137039269a3b8d5365ad92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-10T06:35:11.172 INFO:journalctl@ceph.osd.5.smithi146.stdout:Feb 10 06:35:11 smithi146 podman[80274]: 2024-02-10 06:35:11.118461127 +0000 UTC m=+0.709744144 container died 8e9f2f1fa8b8caa540080af6f8c39836a1a5a06241137039269a3b8d5365ad92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-10T06:35:11.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:11.285+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:11.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:11.285+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:11.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:11.285+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:11.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:11.342+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:11.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:11.342+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:11.673 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:11 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:11.342+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:11.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T06:35:11.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:11.765 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T06:35:11.765 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T06:35:11.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.6 2024-02-10T06:35:12.099 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:11 smithi146 systemd[1]: Stopping Ceph osd.6 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:12.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:12.306+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:12.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:12.306+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:12.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:12.306+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:12.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.098+0000 7ffb3273e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:35:12.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.098+0000 7ffb3273e700 -1 osd.6 840 *** Got signal Terminated *** 2024-02-10T06:35:12.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.098+0000 7ffb3273e700 -1 osd.6 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:35:12.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.303+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:12.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.303+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:12.423 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:12 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:12.303+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:13.603 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:13.266+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:13.603 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:13.266+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:13.604 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:13.266+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:13.604 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:13.276+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:13.604 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:13.276+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:13.604 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:13 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:13.276+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:14.218+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:14.218+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:14.218+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:14.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:14.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:14.534 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:14 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:14.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:15.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:15.264+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:15.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:15.264+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:15.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:15.264+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:15.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:15.254+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:15.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:15.254+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:15.673 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:15.254+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:15.673 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:15 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:15.254+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:16.673 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:16.249+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:28.572633+0000 front 2024-02-10T06:34:28.572597+0000 (oldest deadline 2024-02-10T06:34:52.072466+0000) 2024-02-10T06:35:16.673 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:16.249+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:36.074121+0000 front 2024-02-10T06:34:36.074050+0000 (oldest deadline 2024-02-10T06:35:01.373900+0000) 2024-02-10T06:35:16.673 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:16.249+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:41.374474+0000 front 2024-02-10T06:34:41.374457+0000 (oldest deadline 2024-02-10T06:35:06.074320+0000) 2024-02-10T06:35:16.674 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6[51554]: 2024-02-10T06:35:16.249+0000 7ffb2ed32700 -1 osd.6 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:51.375251+0000 front 2024-02-10T06:34:51.375261+0000 (oldest deadline 2024-02-10T06:35:15.475154+0000) 2024-02-10T06:35:16.674 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:16.301+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:16.674 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:16.301+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:16.674 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:16.301+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:16.675 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:16 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:16.301+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:17.672 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:17 smithi146 podman[80406]: 2024-02-10 06:35:17.471210186 +0000 UTC m=+5.478020633 container died 37e21ccbe6b7eacf56572880028d519630625ac52b3abb6b2f2a6e5db998e239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-02-10T06:35:17.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:17.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:17.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:17.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:17.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:17.267+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:17.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:17 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:17.267+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:18.114 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:17 smithi146 podman[80406]: 2024-02-10 06:35:17.871118897 +0000 UTC m=+5.877929346 container cleanup 37e21ccbe6b7eacf56572880028d519630625ac52b3abb6b2f2a6e5db998e239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T06:35:18.114 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:17 smithi146 bash[80406]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80440]: 2024-02-10 06:35:18.113853501 +0000 UTC m=+0.632091872 container remove 37e21ccbe6b7eacf56572880028d519630625ac52b3abb6b2f2a6e5db998e239 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:18.244+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:18.244+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:18.244+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:18.244+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:18.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:18 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:18.244+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:18.700 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.353235174 +0000 UTC m=+0.019588174 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:35:18.701 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.482012475 +0000 UTC m=+0.148365459 container create 86d0bd90965816a959d982e4cdec28875906edc7762eb86ffb1f15d1a5a5e97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:35:18.993 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.70043738 +0000 UTC m=+0.366790368 container init 86d0bd90965816a959d982e4cdec28875906edc7762eb86ffb1f15d1a5a5e97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T06:35:18.993 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.706105248 +0000 UTC m=+0.372458234 container start 86d0bd90965816a959d982e4cdec28875906edc7762eb86ffb1f15d1a5a5e97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T06:35:18.994 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.800038228 +0000 UTC m=+0.466391222 container attach 86d0bd90965816a959d982e4cdec28875906edc7762eb86ffb1f15d1a5a5e97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T06:35:19.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Feb 10 06:35:18 smithi146 podman[80539]: 2024-02-10 06:35:18.99385873 +0000 UTC m=+0.660211723 container died 86d0bd90965816a959d982e4cdec28875906edc7762eb86ffb1f15d1a5a5e97f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T06:35:19.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.251+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:19.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.251+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:19.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.251+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:19.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.251+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:19.553 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.251+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:19.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T06:35:19.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:19.617 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T06:35:19.618 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T06:35:19.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.7 2024-02-10T06:35:19.922 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 systemd[1]: Stopping Ceph osd.7 for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:20.223 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.947+0000 7fd46e3f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.947+0000 7fd46e3f5700 -1 osd.7 840 *** Got signal Terminated *** 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:19 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:19.947+0000 7fd46e3f5700 -1 osd.7 840 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:20.222+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:20.222+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:20.222+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:20.222+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:20.224 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:20 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:20.222+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:21.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:21.185+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:21.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:21.185+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:21.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:21.185+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:21.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:21.185+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:21.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:21 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:21.185+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:22.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:22 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:22.220+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:22.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:22 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:22.220+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:22.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:22 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:22.220+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:22.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:22 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:22.220+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:22.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:22 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:22.220+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.205+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.206+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.206+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.206+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.206+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-02-10T06:34:58.109606+0000 front 2024-02-10T06:34:58.109747+0000 (oldest deadline 2024-02-10T06:35:22.809716+0000) 2024-02-10T06:35:23.672 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:23 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:23.206+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-02-10T06:34:29.204004+0000 front 2024-02-10T06:34:29.204160+0000 (oldest deadline 2024-02-10T06:34:51.504136+0000) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6814 osd.1 since back 2024-02-10T06:34:37.206292+0000 front 2024-02-10T06:34:37.206263+0000 (oldest deadline 2024-02-10T06:34:58.906182+0000) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6822 osd.2 since back 2024-02-10T06:34:43.408219+0000 front 2024-02-10T06:34:43.408206+0000 (oldest deadline 2024-02-10T06:35:07.508129+0000) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.121:6830 osd.3 since back 2024-02-10T06:34:52.809264+0000 front 2024-02-10T06:34:52.809224+0000 (oldest deadline 2024-02-10T06:35:18.109120+0000) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-02-10T06:34:58.109606+0000 front 2024-02-10T06:34:58.109747+0000 (oldest deadline 2024-02-10T06:35:22.809716+0000) 2024-02-10T06:35:24.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:24 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7[58016]: 2024-02-10T06:35:24.187+0000 7fd4669e1700 -1 osd.7 840 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2215 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e840) 2024-02-10T06:35:25.774 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:25 smithi146 podman[80669]: 2024-02-10 06:35:25.492576567 +0000 UTC m=+5.656854757 container died 0627bc604b374e7a021cf770b0c60e9743931575b9417a13326179056335ecdc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T06:35:26.123 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:25 smithi146 podman[80669]: 2024-02-10 06:35:25.872175036 +0000 UTC m=+6.036453229 container cleanup 0627bc604b374e7a021cf770b0c60e9743931575b9417a13326179056335ecdc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T06:35:26.123 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:25 smithi146 bash[80669]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7 2024-02-10T06:35:26.422 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80691]: 2024-02-10 06:35:26.123061039 +0000 UTC m=+0.631732067 container remove 0627bc604b374e7a021cf770b0c60e9743931575b9417a13326179056335ecdc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T06:35:26.734 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.340300076 +0000 UTC m=+0.017734203 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T06:35:26.734 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.448841057 +0000 UTC m=+0.126275184 container create 5b94647b7f9ecc5d027d99f08fd8337ed458d249ec2d4bf10f610b39061c4e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T06:35:26.734 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.659494511 +0000 UTC m=+0.336928638 container init 5b94647b7f9ecc5d027d99f08fd8337ed458d249ec2d4bf10f610b39061c4e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T06:35:26.734 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.66507982 +0000 UTC m=+0.342513952 container start 5b94647b7f9ecc5d027d99f08fd8337ed458d249ec2d4bf10f610b39061c4e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=) 2024-02-10T06:35:26.734 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.734127798 +0000 UTC m=+0.411561939 container attach 5b94647b7f9ecc5d027d99f08fd8337ed458d249ec2d4bf10f610b39061c4e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:35:27.172 INFO:journalctl@ceph.osd.7.smithi146.stdout:Feb 10 06:35:26 smithi146 podman[80786]: 2024-02-10 06:35:26.958456217 +0000 UTC m=+0.635890343 container died 5b94647b7f9ecc5d027d99f08fd8337ed458d249ec2d4bf10f610b39061c4e6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T06:35:27.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T06:35:27.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:27.594 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T06:35:27.594 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-10T06:35:27.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@rgw.foo.a 2024-02-10T06:35:27.887 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:27 smithi121 systemd[1]: Stopping Ceph rgw.foo.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:28.328 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:27 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a[69665]: 2024-02-10T06:35:27.886+0000 7ff3aed4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T06:35:28.328 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:27 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a[69665]: 2024-02-10T06:35:27.886+0000 7ff3bd5beac0 -1 shutting down 2024-02-10T06:35:38.250 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:37 smithi121 bash[161139]: time="2024-02-10T06:35:37Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-10T06:35:38.250 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:37 smithi121 podman[161139]: 2024-02-10 06:35:37.91033792 +0000 UTC m=+10.124422919 container died 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T06:35:38.532 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:38 smithi121 podman[161139]: 2024-02-10 06:35:38.323405044 +0000 UTC m=+10.537490042 container cleanup 8ba3784c657febd28572411e965c320209ef8591c648a94f24cdb3ac960b2fe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-02-10T06:35:38.533 INFO:journalctl@ceph.rgw.foo.a.smithi121.stdout:Feb 10 06:35:38 smithi121 bash[161139]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-rgw-foo-a 2024-02-10T06:35:38.757 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service' 2024-02-10T06:35:38.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:38.794 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-10T06:35:38.794 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T06:35:38.794 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T06:35:39.089 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:38 smithi146 systemd[1]: Stopping Ceph prometheus.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.088Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.089Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.089Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.090Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T06:35:39.396 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.090Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T06:35:39.397 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a[68695]: ts=2024-02-10T06:35:39.090Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T06:35:39.397 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 podman[80915]: 2024-02-10 06:35:39.099402757 +0000 UTC m=+0.119642797 container died da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:39.672 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 podman[80915]: 2024-02-10 06:35:39.47792175 +0000 UTC m=+0.498161791 container cleanup da4b4b6a9a423784da3281c7837e2a41bf391c6f601005c32d657af530955639 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:39.672 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Feb 10 06:35:39 smithi146 bash[80915]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T06:35:39.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T06:35:39.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T06:35:39.882 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T06:35:39.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:35:40.073 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:35:41.828 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:41 smithi121 systemd[1]: Stopping Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:42.153 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:41 smithi121 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a[80691]: ts=2024-02-10T06:35:41.880Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T06:35:42.154 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:41 smithi121 podman[161646]: 2024-02-10 06:35:41.921982989 +0000 UTC m=+0.146966881 container died ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:42.437 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 podman[161646]: 2024-02-10 06:35:42.236432397 +0000 UTC m=+0.461416282 container cleanup ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:42.437 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 bash[161646]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T06:35:42.786 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 podman[161665]: 2024-02-10 06:35:42.437490586 +0000 UTC m=+0.513635858 container remove ad096e7e4cab39474b5f2b4c3ae9733820f7c7d95fc992c44f8c02396e9e1ecf (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:42.787 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 podman[161665]: 2024-02-10 06:35:42.513039262 +0000 UTC m=+0.589184533 volume remove f6868f902e648f24f58e86ecfe9780dca0daf59df668d9aad94f1ba538421952 2024-02-10T06:35:43.078 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:42 smithi121 systemd[1]: Stopping Ceph node-exporter.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:43.079 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T06:35:43.079 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 systemd[1]: Stopped Ceph alertmanager.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:35:43.079 INFO:journalctl@ceph.alertmanager.a.smithi121.stdout:Feb 10 06:35:42 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 2.545s CPU time. 2024-02-10T06:35:43.516 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:43 smithi121 podman[161780]: 2024-02-10 06:35:43.24603634 +0000 UTC m=+0.121550680 container died bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:43.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:43 smithi121 podman[161780]: 2024-02-10 06:35:43.615165411 +0000 UTC m=+0.490679749 container cleanup bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:43.828 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:43 smithi121 bash[161780]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T06:35:44.205 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:43 smithi121 podman[161798]: 2024-02-10 06:35:43.941189464 +0000 UTC m=+0.695173604 container remove bea7a2d9fbef9495e9ca4eb10fd5d1347bfdd973d2962a325357c3752e6876b9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T06:35:44.206 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:43 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:35:44.206 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:44 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T06:35:44.206 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:44 smithi121 systemd[1]: Stopped Ceph node-exporter.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:35:44.206 INFO:journalctl@ceph.node-exporter.a.smithi121.stdout:Feb 10 06:35:44 smithi121 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 5.264s CPU time. 2024-02-10T06:35:44.680 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T06:35:44.872 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:35:46.172 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:46 smithi146 systemd[1]: Stopping Ceph iscsi.iscsi.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:46.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:46 smithi146 ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[63379]: debug Shutdown received 2024-02-10T06:35:56.657 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:56 smithi146 bash[81338]: time="2024-02-10T06:35:56Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-10T06:35:56.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:56 smithi146 podman[81338]: 2024-02-10 06:35:56.369927232 +0000 UTC m=+10.135204181 container died a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:35:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:56 smithi146 podman[81338]: 2024-02-10 06:35:56.738809984 +0000 UTC m=+10.504086933 container cleanup a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T06:35:56.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:56 smithi146 bash[81338]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a 2024-02-10T06:35:57.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:57 smithi146 podman[81375]: 2024-02-10 06:35:57.056610586 +0000 UTC m=+0.687219914 container remove a2c48fd974cb3db4ba87d0e6967edd781be3fdeecc76c15a04c7a97c90b3adfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T06:35:57.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:57 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T06:35:57.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:57 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-10T06:35:57.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:57 smithi146 systemd[1]: Stopped Ceph iscsi.iscsi.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:35:57.353 INFO:journalctl@ceph.iscsi.iscsi.a.smithi146.stdout:Feb 10 06:35:57 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Consumed 4.468s CPU time. 2024-02-10T06:35:58.852 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:58 smithi146 systemd[1]: Stopping Ceph grafana.a for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:59.172 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:58 smithi146 podman[81651]: 2024-02-10 06:35:58.85257921 +0000 UTC m=+0.111928851 container died 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, name=ubi8, release=239.1651231664) 2024-02-10T06:35:59.584 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 podman[81651]: 2024-02-10 06:35:59.179957766 +0000 UTC m=+0.439307416 container cleanup 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, com.redhat.component=ubi8-container, distribution-scope=public, architecture=x86_64, maintainer=Paul Cuzner , name=ubi8, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.openshift.tags=base rhel8) 2024-02-10T06:35:59.584 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 bash[81651]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a 2024-02-10T06:35:59.584 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 podman[81670]: 2024-02-10 06:35:59.372710639 +0000 UTC m=+0.519225949 container remove 9a46ca9fc38d14a5c9f7c115fe211b0841d326504d6e2fc4232f920ac05b6f1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, vcs-type=git, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.component=ubi8-container, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64) 2024-02-10T06:35:59.584 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:35:59.922 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:35:59 smithi146 systemd[1]: Stopping Ceph node-exporter.b for 616c67f8-c7d8-11ee-95b8-87774f69a715... 2024-02-10T06:35:59.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T06:35:59.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 systemd[1]: Stopped Ceph grafana.a for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:35:59.922 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Feb 10 06:35:59 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Consumed 18.345s CPU time. 2024-02-10T06:36:00.334 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 podman[81787]: 2024-02-10 06:36:00.055129613 +0000 UTC m=+0.113407328 container died dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:36:00.672 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 podman[81787]: 2024-02-10 06:36:00.433414961 +0000 UTC m=+0.491692684 container cleanup dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:36:00.672 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 bash[81787]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T06:36:00.947 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 podman[81806]: 2024-02-10 06:36:00.676104453 +0000 UTC m=+0.622064021 container remove dde9e89e9bdf9b3368dbe7b874c601f5e84525353ae31a9d96a4300af485ad26 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-616c67f8-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T06:36:00.947 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T06:36:00.947 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T06:36:00.947 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 systemd[1]: Stopped Ceph node-exporter.b for 616c67f8-c7d8-11ee-95b8-87774f69a715. 2024-02-10T06:36:00.947 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Feb 10 06:36:00 smithi146 systemd[1]: ceph-616c67f8-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 5.540s CPU time. 2024-02-10T06:36:01.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:01.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T06:36:01.455 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T06:36:01.457 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi121/crash 2024-02-10T06:36:01.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:36:01.518 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:36:01.519 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:36:01.520 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi146/crash 2024-02-10T06:36:01.521 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T06:36:01.574 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T06:36:01.574 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-02-10T06:36:01.575 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T06:36:01.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-02-10T06:36:01.607 INFO:teuthology.orchestra.run.smithi121.stdout:2024-02-10T06:10:00.194224+0000 mon.a (mon.0) 2885 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-02-10T06:36:01.608 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T06:36:01.609 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[SEC\]' /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-02-10T06:36:01.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]' /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-02-10T06:36:01.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[WRN\]' /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-02-10T06:36:01.824 INFO:teuthology.orchestra.run.smithi121.stdout:2024-02-10T06:10:00.194224+0000 mon.a (mon.0) 2885 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-02-10T06:36:01.825 INFO:tasks.cephadm:Compressing logs... 2024-02-10T06:36:01.825 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:36:01.867 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T06:36:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:36:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2024-02-10T06:36:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T06:36:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:36:01.893 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-10T06:36:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T06:36:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T06:36:01.899 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T06:36:01.899 INFO:teuthology.orchestra.run.smithi146.stderr:: No such file or directory 2024-02-10T06:36:01.899 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T06:36:01.899 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log: 67.5% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:36:01.900 INFO:teuthology.orchestra.run.smithi146.stderr: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:36:01.901 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:36:01.901 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T06:36:01.901 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T06:36:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T06:36:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T06:36:01.902 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T06:36:01.905 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: 87.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:36:01.906 INFO:teuthology.orchestra.run.smithi146.stderr: 92.2%/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log: -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:36:01.908 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T06:36:01.909 INFO:teuthology.orchestra.run.smithi146.stderr: 92.7% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T06:36:01.911 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T06:36:01.911 INFO:teuthology.orchestra.run.smithi121.stderr: 95.2% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T06:36:01.912 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T06:36:01.913 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:36:01.922 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log: 93.3%gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T06:36:01.923 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T06:36:01.923 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T06:36:01.938 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T06:36:01.968 INFO:teuthology.orchestra.run.smithi121.stderr: 94.3% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-10T06:36:02.470 INFO:teuthology.orchestra.run.smithi121.stderr: 90.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T06:36:02.962 INFO:teuthology.orchestra.run.smithi146.stderr: 92.8% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T06:36:03.615 INFO:teuthology.orchestra.run.smithi121.stderr: 92.4% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T06:36:04.381 INFO:teuthology.orchestra.run.smithi121.stderr: 91.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T06:36:09.780 INFO:teuthology.orchestra.run.smithi146.stderr: 93.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T06:36:10.103 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T06:36:10.405 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T06:36:10.513 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T06:36:10.689 INFO:teuthology.orchestra.run.smithi121.stderr: 93.8% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T06:36:10.757 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T06:36:10.758 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-02-10T06:36:10.759 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m8.877s 2024-02-10T06:36:10.759 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m33.362s 2024-02-10T06:36:10.759 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m1.178s 2024-02-10T06:36:10.876 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T06:36:11.197 INFO:teuthology.orchestra.run.smithi121.stderr: 94.0% -- replaced with /var/log/ceph/616c67f8-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T06:36:11.199 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-02-10T06:36:11.199 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m9.320s 2024-02-10T06:36:11.199 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m39.075s 2024-02-10T06:36:11.199 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m1.340s 2024-02-10T06:36:11.199 INFO:tasks.cephadm:Archiving logs... 2024-02-10T06:36:11.199 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi121/log 2024-02-10T06:36:11.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:36:16.684 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi146/log 2024-02-10T06:36:16.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T06:36:21.765 INFO:tasks.cephadm:Removing cluster... 2024-02-10T06:36:21.765 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T06:36:21.957 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:36:22.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 616c67f8-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T06:36:22.437 INFO:teuthology.orchestra.run.smithi146.stdout:Deleting cluster with fsid: 616c67f8-c7d8-11ee-95b8-87774f69a715 2024-02-10T06:36:22.728 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T06:36:22.729 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:36:22.744 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T06:36:22.759 INFO:tasks.cephadm:Teardown complete 2024-02-10T06:36:22.760 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T06:36:22.769 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T06:36:22.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T06:36:22.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T06:36:22.875 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T06:36:22.886 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T06:36:22.886 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:36:22.889 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T06:36:22.900 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:36:22.902 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-02-10T06:36:22.905 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:36:22.905 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-02-10T06:36:22.905 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 87 +694us[ +687us] +/- 79ms 2024-02-10T06:36:22.905 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 7 377 84 -1347us[-1354us] +/- 77ms 2024-02-10T06:36:22.906 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +872us[ +872us] +/- 80ms 2024-02-10T06:36:22.906 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:36:22.906 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 20 +687us[ +681us] +/- 80ms 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -1359us[-1365us] +/- 78ms 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +861us[ +855us] +/- 80ms 2024-02-10T06:36:22.907 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T06:36:22.907 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T06:36:22.918 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T06:36:22.919 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T06:36:22.928 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:36:23.026 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T06:36:23.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:36:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:36:23.135 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T06:36:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:36:23.166 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T06:36:23.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T06:36:23.186 INFO:teuthology.task.internal:Duration was 3122.960163 seconds 2024-02-10T06:36:23.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T06:36:23.196 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T06:36:23.196 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:36:23.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T06:36:23.240 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:36:23.250 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T06:36:23.616 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T06:36:23.616 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-02-10T06:36:23.616 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:36:23.675 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-02-10T06:36:23.676 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T06:36:23.700 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T06:36:23.701 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:36:23.718 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T06:36:23.770 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T06:36:23.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:36:23.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T06:36:24.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T06:36:24.237 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T06:36:24.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:36:24.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T06:36:24.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T06:36:24.307 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:36:24.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:36:24.333 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-02-10T06:36:24.357 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-02-10T06:36:24.389 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:36:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:36:24.403 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T06:36:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T06:36:24.444 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T06:36:24.454 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T06:36:24.455 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi121 2024-02-10T06:36:24.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:36:24.568 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098/remote/smithi146 2024-02-10T06:36:24.568 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T06:36:24.609 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T06:36:24.609 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:36:24.611 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T06:36:24.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T06:36:24.668 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T06:36:24.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T06:36:24.679 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T06:36:24.679 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:36:24.682 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T06:36:24.693 INFO:teuthology.orchestra.run.smithi121.stdout: 265595 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:36 /home/ubuntu/cephtest 2024-02-10T06:36:24.712 INFO:teuthology.orchestra.run.smithi146.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 06:36 /home/ubuntu/cephtest 2024-02-10T06:36:24.713 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T06:36:24.725 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T06:36:24.751 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T06:36:24.775 DEBUG:teuthology.nuke:shortname: smithi121 2024-02-10T06:36:24.776 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:36:24.796 DEBUG:teuthology.nuke:shortname: smithi146 2024-02-10T06:36:24.797 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T06:36:24.800 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:35:29.839966', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJDSuDyeUMsUBlYzLQH3vS26Qj+6VwA00H7V7MCI3BzTPM51SbgVUPYfKIj1G45pWjrQRmQZuJ7e7YTEvf85GKY='} 2024-02-10T06:36:24.818 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555098', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:35:29.838543', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBETJdK0fND4MHZCqDjkm30lfLE4kzHagSTV9vkePHnOc9S9KQxA7OCEnMWhnDF2ewnFfpA68J3cfkJpZXoE51E='} 2024-02-10T06:36:24.848 INFO:teuthology.orchestra.console.smithi121:Power off 2024-02-10T06:36:24.848 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:36:24.874 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-02-10T06:36:24.874 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:24.889 INFO:teuthology.orchestra.console.smithi146:Power off 2024-02-10T06:36:24.890 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T06:36:24.899 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-10T06:36:24.912 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-02-10T06:36:24.913 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:24.938 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-02-10T06:36:28.900 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:28.939 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:29.010 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-10T06:36:29.049 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-02-10T06:36:33.016 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:33.050 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:33.126 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-02-10T06:36:33.159 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-02-10T06:36:33.260 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-02-10T06:36:37.129 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T06:36:37.239 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-02-10T06:36:37.340 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-02-10T06:36:37.444 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} duration: 3122.9601633548737 failure_reason: '"2024-02-10T06:10:00.194224+0000 mon.a (mon.0) 2885 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-10T06:36:37.445 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T06:36:37.523 INFO:teuthology.run:FAIL